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ABSTRACT 

This learning module, one in a series of 
competency-based guidance program training packages focusing upon 
professional and paraprof essional competencies of guidance personnel, 
deals with evaluating guidance activities. Addressed in the module 
are the following topics: writing evaluation questions, preparing an 
evaluation plan, selecting evaluation strategies and instruments, 
collecting and analyzing data, and evaluating an evaluation. The 
module consists of readings and learning experiences covering these 
five topics. Each learning experience contains some or all of the 
following: an overview, a competency statement, a learning objective, 
one or more individual learning activities, an individual feedback 
exercise, one or more group activities, one or more handouts, and a 
facilitator's outline for use in directing the group activities. 
Concluding the module are a participant self-assessment 
questionnaire, a trainer's assessment questionnaire, a checklist of 
performance indicators, a list of references, and an annotated list 
of suggested additional resources. (MN) 
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FOREWORD 



rim (OtirrtMtnq and guidance program series rs patterned 
alter the Performance Batted Teacher Educaiion modules 
designed and developed at Ihe National Center lor Research 
in Vocalronai Education under Federal Number NE-COO-3-77 
Because this model has been successfully and enthusiastically 
recrevednaiionaily and internationally this series of modules 
loliows the same basic format 

This module is one of a series oi competency-based guidance 
program training packages focusing upon specific professional 
and paraprotessional competencies ot guidance personnel' 
The competencies upon which these modules ue based were 
identified and verified through a project study as being those 
of critical importance tor the planning supporting implement- 
ing operating and evaluating of guidance programs These 
modu>es are addressed to professional and paraprotessional 
guidance program staff in a wide variety ot educational and 
community sellings and agencies 

Each module provides learning experiences that integrate 
theory and application each culminates with competency 
relerented evaluation suggestions The materials are designed 
tor use by individuals or groups of guidance personnel who 
are involved in Iratnmg Resource persons should be Skilled in 
the guidance program competency being developed and 
should be thoroughly oriented tothe concepts and procedures 
used in the total framing package 

The design of the materials provides considerable flexibility 
tor planning and conducting competency-based preservice 
and inservice programs to meet a wide variety ot individual 
needs and mierests The materials are intended tor use by 
universities state departments of education, postsecondary 
institutions intermediate educational service agencies, JTPA 
agencies employment security agencies and other commu- 
nity agencies thai are responsible tor the employment and 
professional development ot guidance personnel 

The competency-based guidance program training packages 
are products of a research effort by the National Center s 
Career Development Program Area Many individuals institu- 
tions and agencies participated with the National Center and 
have made contributions to Ihe systematic development 
testing and refinement of the materials 



National consultants provided substantial writing and review 
assistance in development ot Ihe initial module versions over 
1 300 guidance personnel used the materials in early stages ot 
their development and provided feedback fo the National 
Center tor revision and refinement The materials have been 
or are being used by 57 pilot community implementation sites 
across the country 

Special recognition tor major roles in the direction de we lop- 
men I coordination of development, testing and revision ot 
these materials and the coordination of pilot implementation 
sites is extended to the following protect staff Harry N Drier 
Consortium Director. Robert E Campbell Linda PfiSier 
Directors, Robert Bhaerman. Research Specialist Karen 
Kimmel Be yle. Fred Williams Program Associates, and Janie 
B Connell, Graduate Research Associate 

Appreciation afso is extended to the subcontractors who 
assisted Ihe National Center in thisettort Drs Brian Jones 
and Linda Phillips-Jones ot ihe American Institutes for 
Research developed the competency base for the total pack- 
age managed protect evaluation, and developed the modules 
addressing special needs Gratitude is expressed lo Dr 
Norman Gysbers of the University of Missouri-Columbia for 
his work on the module on individual career development 
plans Both ot these agencies provided coordination and 
monitoring assistance for the pilot implementation sites 
Appreciation is extended to the American Vocational Associ- 
ation and the American Association tor Counseling and 
Development tor their leadership in directing extremely impor- 
tant subcontractors associated with the first phase ot this 
effort 

The National Center *s grateful to the U S Department of 
Education, OtticeotVocationatand Adult Education (OVAE) 
for sponsorship ot three contracts retaled to thiscompetency- 
ba&ed guidance program training package In particular we 
appreciate the lea dershipand support offered proiecistatt by 
Dfivid H Pntchard who served as fhe project officer for the 
contracts We feel the investment ot the OVAE in this framing 
package is sound and will have lasting effects in the field ot 
guidance in ihe years to come 

Robert £ Taylor 
Executive Director 
National Center tor Research 
in Vocational Education 
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The National Center for Research m Vocational Education s mis- 
sion is to .ncrease the ability oi diverse agencies, institutions, and 
or g«*n i nations io solve educational problems t elating lo individual 
< *iree* planning preparation, and progression The Nalional Center 
i. hi. Ns its mission by 

* Genetalmg knowledge 1h rough research 

* Developing educational programs and products 

* Evaluating indrvjdual program needs and outcomes 

* Providing information for national planning and policy 

* installing educational programs and products 

* Operating information systems and services 

* Conducing leadership development and training pfograms 



Bell & HowelF, Publication Products, is one of two operating units 
that comprise Publication Systems Division Based m Wooster, 
Ohio, Publication Products specializes in Ihe production and repro- 
duction ol newspapers, periodicals, rndeaes, cateer information 
materials and other widely used information sources in microform, 
hard copy and electronic media 
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ABOUT THIS MODULE 



GOAL - ■ ,.<. 

Aft«r competing tht« nf)pdul». th« o$rwr fluldince 
prpori^ 

EVALUATE GUIDANCE 

activity* ^^^^^^^^ ^ 

indtvWUtl program comp<>r^ti »rwt d«l#fmlntng 
th# d*gret td whitri th*|ir^^ 
goal* in a coefficient manner. 

INTRODUCTION 5 

READING 7 

Competency 1. Verity the relationship of the components of the activity to be assessed. 

including statements ol goats, ctient performance objectives, and process objectives 9 

Competency 2. Select or develop an appropriate evaluation design that includes a 
comparison standard tor determining it gains made by the clients can be attributed to 
the program 9 

Competency 3. Select criteria tor evaluation procedures and instruments, and adapt, adopt. 

or create measures ot the attainment ot program process and performance objectives 11 

Competency 4. Select criteria tor evaluation instruments, and select, adapt, or create 

appropriate evaluation instruments and strategies 11 

Competency 5. Administer the evaluation instruments according to the evaluation design, 
and analyze and display the evaluation data in a usable form 13 

Competency 6. Apply the selected evaluation measures to assess the attainment ot program 

process and performance objectives 14 

Competency 7 Evaluate the process used in the formative evaluation of the program to make 
necessary alterations in this process 16 

LEARNING EXPERIENCES 

1 Writing Evaluation Questions 17 

2 Preparing the Evaluation Plan 23 

3. Selecting Evaluation Strategies and Instruments 31 

4 Collecting and Analyzing the Data 43 

5 Evaluating the Evaluation 49 

EVALUATION 71 

REFERENCES 79 
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ABOUT USING THE CBCG MODULES 



CBCQ Module Organization 

The training modules cover the knowledge, skills, and 
attitudes needed to plan, support, implement, operate, and 
evaluate a comprehensive career guidance program. They 
are designed to provide career guidance program im- 
plemented with a systematic means to improve their 
career guidance programs. They are competency-based 
and contain specific information that is intended to assist 
users to develop at least part of the critical competencies 
necessary for overall program improvement. 

These modules provide information and learning ac- 
tivities that are useful tor both school-based and 
nonschool*based career guidance programs. 

The modules are divided into five categories. 
The GUIDANCE PROGRAM PLANNING category assists 
guidance personnel in outlining in advance what is to be 
done. 

The SUPPORTING category assists personnel in know- 
ing how to provide resources or means that make it possi- 
ble for planned program activities to occur. 
The IMPLEMENTING category suggests how to conduct* 
accomplish* or carry out selected career guidance program 
activities. 

The OPERATING category provides information on how 
to continue the program on a day-to-day basis once it has 
been initiated, 

The EVALUATING category assists guidance personnel 
in judging the quality and Impact of the program and either 
making appropriate modifications based on findings or 
making decisions to terminate it. 

Module Format 

A standard format is used in all of the program's 
competency-based modules. Each module contains (1) an 
introduction, (2) a module focuj, (3) a reading, (4) learn- 
ing experiences, (5) evaluation techniques* and (6) 
resources, 

Introduction. The introduction gives you, the module 
user an overview of the purpose and content of the 
module. It provides enough Information for you to deter- 
mine if the module addresses an area in which you 
need more competence, 

About Thte Module. This section presents the follow- 
ing information: 

Module Goal: A statement of what one can ac- 
complish by completing the module. 
Competencies: A listing of the competency 
statements that relate to the module's area of con- 
cern. These statements represent the competencies 
thought to be most critical in terms ot difficulty for 
inexperienced implementerst and they are not an 
exhaustive list. 

This section also serves as the table of contents for tne 
reading and learning experiences, 

Reading. Each modul* contains a section in which 
cognitive information on each one of the competencies 
is presented. 

1 . Use it as a te xtbook by starting at the first page and 
reading through until the end. You could then 



ERIC 



complete the learning experiences that relate to 
specific competencies. This approach is good if you 
would like to give an overview of some competen- 
cies and a more irnlepth study of others. 

2. Turn directly to the learning experiences^) that 
relate to the needed competency (competencies). 
Within each learning experience a reading Is listed, 
This approach allows for a more experiential ap* 
proach prior to the reading activity. 

Learning Experiences. The learning experiences are 
designed to help users in the achievement of specific 
learning objectives. One learning experience exists for 
each competency (or a duster of like competencies), and 
each learning experience is designed to stand on its own. 
Each learning experience is preceded by an overview 
sheet which describes what is to be covered fn the learn* 
ing experience. 

Within the body of the learning experience* the following 
components appear. 

Individual Activity: This is an activity which a person 
can complete without any outside assistance. All of the 
information needed for its completion is contained in 
the module. 

individual Feedback: After each individual activity 
there is a feedback section. This is to provide users 
with immediate feedback or evaluation regarding their 
progress before continuing. The concept of feedback 
is also intended with the group activities, but it is built 
right into the activity and does not appear as a separate 
section. 

Group Activity: This activity is designed to be 
facilitated by a trainer, within a group training session. 

The group activity is formatted along the lines of a 
facilitator's outline. The outline details suggested ac- 
tivities and information for you to use, A blend of 
presentation and "hands-on* participant activities such 
as games and role playing is included. A Notes column 
appears on each page of the facilitator's outline. This 
space is provided so trainers can add their own com- 
ments and suggestions to the cues that are provided. 

Following the outline Is a list of materials that will be 
needed by workshop facilitator. This section can serve 
as a duplication master for mimeographed handouts 
or transparencies you may want to prepare. 

Evaluation Techniques* This section of each module con- 
tains information and instruments that can be used to 
measure what workshop participants need prior to train* 
ing and what they have accomplished as a result of train- 
ing, included in this section are a Pre* and Post*Participant 
Assessment Questionnaire and a Trainer's Assessment 
Questionnaire, The latter contains a set of performance 
indicators which are designed to determine the degree of 
success the participants had with the activity. 

References. All major sources that were used to develop 
the module are listed in this section, Alsa major materials 
resources that relate to the competencies presented in the 
module are described and characterized. 



INTRODUCTION 



Evaluation is concerned with judging the ade- 
quacy or worth of something. It is a management 
tool *nat you can use to improve your career guid- 
ance program Your career guidance program 
evaluation should have two principalfocuses; (1) 
formative evaluation activities-evaluating indi- 
vidual program components on an ongoing basis 
to determine the degree to which they are meet- 
ing client objectives-and (2) summative evalua* 
Hon activities-following up clients to determine 
the overall worth of the total program in achieving 
its goals. The former will provide you with infor- 
mation useful in attaining and maintaining high 
levels of productivity for the individual compo- 
nents of your career guidance program. The lat- 
ter will provide you with information useful in 
identifying the strength sand weaknesses of your 
overall program of services to clients, 

Evaluation does not mean some "outsider" or 
"snoopervisor" keeping watch to see whether or 
not you are succeeding with the clients you seek 
to serve You and some of your colleagues can 
have the necessary training and experience in 
formative and summative evaluation and be able 
to design and carry out an evaluation plan with 
the remaining career guidance staff. Ideally, those 
in your setting who are responsible for develop- 
ing and providing careerguidanceservicesare in 
the best position to monitor and evaluate your 
program s processes and products and to collect 
and present to the decision makers data needed 
to improve your client services. However* you 



may find it desirable to employ an "outsider' 1 to 
assist you with your program evaluation activi- 
ties. For example, if you and your colleagues have 
neither the time or the expertise to work effec- 
tively in such areas as selecting or developing 
instruments or collecting and analyzing data, you 
may want to have an external evaluator assist you 
with these tasks. Moreover, if you decide to use 
the services of an external evaluator, that person 
should be involved from the very beginning of the 
planning stages and throughout the program 
development and implementation stages. 

The purpose of this module is to assist you in 
developing some of the competencies needed to 
plan, carry out, and evaluate both formative and 
summative evaluation activities to improve your 
career guidance program. It is a key module in 
this training package. This module wilt help you 
the most in identifying your successes as you 
introduce into your career guidance program the 
program changes being suggested by all these 
modules. As you complete this module, please 
remember that the competencies and skills it 
seeks to assist you in developing are not the only 
ones you will need in order to plan implement 
and evaluate adequate formative and summative 
evaluation activities for your career guidance 
program. This module will, however, provide you 
with the information and skills most needed to 
become a good evaluator of the career guidance 
activities offered by your setting. 



READING 

Getting Started 



Evaluation activities should not be considered as 
ends in themselves. They can and do serve many 
purposes-the selection of scholarship winners, 
the determination of who needs special client 
services, the certification of skilled workers, 
among many others. In all these uses* there are 
two common elements. One, you conduct evalua- 
tion activities for decision-making purposes and, 
therefore, you must be responsive to the informa- 
tion needs of the decision makers. Two, you must 
evaluate against some specified criteria or set of 
standards. 

As you design and carry out evaluation activities 
toimproveyourcareerguidance program, remem- 
ber they must be based upon theelementsconsid- 
ered important by the decision makers— your 
program managers or administrators. Generally, 
the important elements will be those elements 
that will help them judge your program's effi- 
ciency, effectiveness, and costs. Examples include 
numbers and types of clients served, types of 
activities and services, and their impact and 
costs. However, the data and information asso- 
ciated with these elements will not be significant 
in themselves. There is no way for your program 
managers or administrators to attach meaning to 
them unless they have some standards or criteria 
for judgment that permit them to make useful 
comparisons, and, ultimately, evaluative judg- 
ments based on these comparisons. 

Evaluation of your career guidance program is 
occurring right now Unfortunately, due to lack of 
time and resources (and a host of other good 
reasons), much of it may be on a subjective level 
and characterized by hit-and-miss methods. Often 
it is based on intuition, not well organized, and 
perhaps even wasteful. The purpose of this module 
is to assist you in developing and implementing 
systematic, conscious, and useful methods and 
procedures that you can use to obtain answers to 
(he following questions: 

1 Is our career guidance program meeting the 
career development needs of our clients? 

2 Are our client records accurate and 
complete 7 



3. Is our testing program broad enough in 
scope? 

4. Does our job placement program function 
effectively? 

5. <s our new career resource center ade- 
quately serving our clients* needs forcareer 
information? 

The bottom line is, carefully designed evaluation 
activities can provide data and information that 
you can use to determine if— 

1, the service provided to your clients is in 
accordance with both the objective of that 
service and the overall goals of the career 
guidance program, 

2* the service(s) being provided to your clients 
should be altered in order to better achieve 
the objective, and 

3, the objective should be altered to bring it 
more in line with the perceived outcomes of 
the service(s) as expressed by your clients. 

Finally, you must use the results of your evalua- 
tion activities. The results or findings must be 
analyzed and disseminated if they are going to be 
available foryouruse in improving your program. 
Your responsibility at this stage in the evaluation 
process cannot be emphasized too strongly. 



Starting the Activity 

Prior to initiating an evaluation of your career 
guidance program, you should meet with your 
co-workers who are to be involved in the activity. 
The basic purpose of this meeting is for you to 
discuss and formalize with other staff members 
the procedures for implementing the activity. In 
addition, it is an excellent time foryou to discuss 
with other staff members the value of this activity 
and how important they are to its success. The 
following items are ones that you should discuss 
very thoroughly with your co-workers, 
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1 Thereaton(t)for evaluating thecareer guid- 
ance program 

• Your board of control asks for it 

• You and some of your colleagues recog- 
nize some weaknesses and concerns 

• You are preparing to receive an on-site 
visitation from an accrediting agency 

• Your current and former clients report 
general dissatisfaction with certain pro- 
gram activities and services 

2 The program process and performance objec- 
tives for the various activities and services 
you provide to your clients 

• Your career resource center 

• Your job placement service 

• Your assistance to parents 

• Your group guidance instruction 

3 Formulation of key questions that focus on 
(a) the program process and (b) client per- 
formance objectives for those program com* 
ponents you are seeking to evaluate 

4. The possible sources of data and informa- 
tion that you may find useful in answering 
each key question 

• Clients (present and past) 

• Staff 

• Parents, spouses* guardians 

• Employers, supervisors 

• Advisory committee members 

• Program staff 

5 The possible ways you can obtain pertinent* 
valid, and reliable information and data 

• Intervtews 

• Surveys (mail* telephone) 

• Testing 



• Unobtrusive measures 

• Review of records 

6- The tasks and procedures to be included in 
your evaluation plan 

• Appointing a coordinating team 

• Establishing a timetable for the evaluation 

• Deciding on program areas tobe evaluated 

• Obtaining administrative approvals 

• Determining an evaluation design 

• Collecting data 

• Summarizing data 

• Reporting results 

• Utilizing the results 



Appointing the Evaluation Team 

Once you have made the decision to evaluate 
your career guidance prog ram* you should appoint 
an evaluation team or task group. It should be 
charged with the overall responsibility of plan- 
ning, coordinating, and evaluating the evaluation 
activities. If you plan to evaluate several program 
components (e.g. client appraisal and records, 
individual and group guidance services* place- 
ment services* the career resource center and 
the guidance facilities), you may want to appoint 
some small committees to assist the evaluation 
team* sach to be responsible for an individual 
program component. Of course* the depth and 
scope of the evaluation and the size of your 
agency or institution would help determine the 
feasibility of forming and involving these small 
committees. If you do decide to use them* it is a 
good idea for each to be chaired by a member of 
the original evaluation team. This strengthens 
considerably both the control and communica- 
tions factors. 

The persons you select to serve on both the evalu- 
ation team and the smaller support committees 
should include representation from among the 
guidance staff* teachers, and administrators asso- 
ciated with your program; present and former 
clients (including "dropouts" or those you were 




less than successful with); members of your advi- 
sory board or committee; aswell as local employ- 
ers and representatives from local service agen- 
cies who "know" the clientele your program 
should be serving. Remember, when you select 
these people, pick not only persons who can con- 
tribute to but those who can benefit from the 
activity as well By exposing key persons to the 
excellent things your program is doing, you can 
gain much local supportforyourcareerguidance 
program. It will come through their participation 
in your evaluation activity. Finally, you should 
charge the evaluation team with the responsibil- 
ity of developing and carrying out the following 
evaluation tasks. 

t Determining the program components to be 
evaluated 

2. Formulating the evaluation questions to be 
answered tor each component 

3. Finalizing the evaluation design and an 
appropriate time-action plan 



4. Selecting, modifying, or developing appro- 
priate measurement instruments 

5. Collectiogdata needed to answer the evalu- 
ation questions 

6. Analyzing the data collected 

7. Reporting the findings of the evaluation 
activities 



Obtaining Approvals 

The team should begin by obtaining expressions 
of support and authorization from the chief admin* 
istrator(s) of the institution or agency responsible 
for your career guidance program. This support 
is essential if your team is tofind support for its 
efforts among those whose support and coopera- 
tion are necessary to the evaluation. You should 
also make sure that the budget (financial support 
aswell as released time) is adequate* understood, 
and acceptable to all concerned. 



Preparing the Evaluation 



Competency 1 



Verify the rtrtailoriih^ of tha comportfnte 
activity to ba assessed, including 
goats, client performance 
objectives. 




of lb* 
AH of 



Competency 2 



Select or develop an appropriate evaluation design 
that Includes * comparison standard tor determin- 
ing if galna made by the client* can be attributed to 
the program. 



The team has several decisions to make in the 
process of preparing the evaluation pian, The 
team needs to do the following: 

1 Define precisely what It wants to learn about 
the career guidance program and the ser- 
vices being provided to clients. The number 
of program components or areas of concern 



to be assessed and the depth of the evalua- 
tion for each will be determined more by 
your information needs, the availability of 
budget, personnel, and the time available 
than by other factors. The primary focus of 
the evaluation plan, however, will be deter- 
mined by your team's concern for formative 
evaluation (assessing individual program 
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components on an ongoing basis) or sum- 
mative evaluation (assessing client's growth 
and development as a result of the pro* 
gram). It is hoped that the team will be inter- 
ested in including both formative and sum- 
mative evaluation activities in the plan. 

2 Develop broad key questions thai reflect the 
overall purposes of the areas of concern it 
decides to include In the evaluation. For 

example, if the career resource center is 
selected as an area of concern for the evalu- 
ation and the team is interested in determin- 
ing whether or not the center's program, 
facilities, and services are of value to clients, 
one key question might be "what is the 
scope, availability, utilization, and effective- 
ness of the client services provided by the 
career resource center?" Stated in more 
simple terms, you might pose the question 
"to what degree are the objectives of the 
career resource center consistent with those 
of your career guidance program and to 
what degree are they being attained?'* The 
foregoing types of key questions would be 
oriented toward formative evaluation or 
assessment of selected process components 
of the career guidance program. 

The team also should consider posing some 
summative evaluation key questions that 
focuson thecareerguidanceprogram'send 
products— the growth experienced by the 
clients it seeksto serve. For example, assume 
that client performance objectives are se- 
lected as an area of concern and that the 
team is interested in determining the degree 
to which client performance objectives which 
address assisting clients to develop toward 
productive and rewarding careers are being 
attained. One client performance-oriented 
key question might be "To what degree are 
ourcltents knowledgeable about thegeneral 
characteristics of occupational 'ields and 
the general training requirements for differ- 
ent occupational levels?" Another client 
performance key question might be 'To 
what degree are each of our clients assum- 
ing responsibility for the direction of their 
own career?' 1 Many key questions can be 
asked Determine which are the important 
questtons. what decisions can be made 
when you have the data needed to answer 
these questions, and how you can best go 
about collecting the needed data, 



3. Select design and analysis alternatives that 
are appropriate for the key evaluation ques- 
tions included in the evaluation* The most 
appropriate design for a given key question 
will depend on— 

• sources of information and instrumenta- 
tion available to you; 

• your local constraints including costs, 
time, personnel, and your access to infor- 
mation sources; and 

• level of precision you desire in answering 
the question—rigorous proof, reasonably 
informed assessment, or something in 
between. 

This latter point will be particularly impor- 
tant to you if you are interested in answering 
with precision summative evaluation ques* 
tions such as ''Can the growth we see in our 
clients be attributed to our program? 1 ' "Does 
our program (or a selected portion of it) 
make a difference in our clients' growth and 
development?*' 

4, Decide on methods and techniques for ob- 
taining the Information needed to answer 
both the formative and summative key ques- 
tions* You may decide to apply some guide- 
lines or standards prepared by others for 
use in measuring the adequacy of selected 
components of a career guidance program. 
Evaluative Criteria for a Secondary School 
Guidance Program published by the Ohio 
Department of Education, Division of Guid- 
ance and Testing, is an example of available 
guidelines. Though this document is school 
focused, many of the guidelines presented 
are adaptable to a variety of career guidance 
program settings. 

If suitable guidelines are not available to 
coverall program componentsto be included 
in the evaluation, then you will need to use 
some other research methods. Use one or 
more questionnaires. They are an efficient 
method of obtaining evaluative information. 
You can administer them to current clients 
and staff members in minimal amounts of 
time, and you can mail them to former clients 
and other information sources including 
parents and employers at relatively low 
costs. 
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Alsn us! - interviews si method of obtaining 
information Inose can b« conducted in 
person Often, a considerable amount of 
trmo ;incf travel expense can be saved r* you 
conduct interviews over ih« telephone 

F iridlly. consider using unobtrusive mea- 
sures Data on your clients" attendance, 
their use of available services and achieve- 
ment of program objectives, referrals re- 
ceived and program costs can be useful in 
answering the key questions included in 
your evaluation design 



In any event, yuur team should use several 
techniques for gathering data Multiple mea- 
surements and data collected by several 
techniques are fi'vely to produce: greater 
confidence m the findings of the evaluation 

Once the evaluation plan is completed, it will pro- 
vide the direction needed >o implement your pro- 
gram evaluation activities 



Preparing Evaluation Strategies and Instruments 



ConiDfitencv 3 nt* and adatrt; adopt, or create measures pf the 

K y attainment hf program process and performance 

objectives* v 



Select criteria for evaluation instruments, and select. 
Competency 4 adapt Or creat^tipprpprlate evaluation instruments 

> and strategies, - 



Quite often standardized instruments that will 
serve all your measurement needs are not avail- 
able You will need to adapt existing or develop 
now instruments The following are stops you 
should complete wh^n not using standardised 
instruments 

1 Develop the support or criterion questions 
tor each of ihe broad key questions. As 

stntpri enrlier, one of your key questions 
nught be Whnt is vhe scope, availability, 
utilization, and effectiveness of the chonl 
servers provided by the rprerr resource 
nwiter^' Ohvicusly. there are many sub- 
rj nest ions you might want to answer in hvalu- 
r^tnu) thesr aspects of the client scr^ccs 
piov^i^d by the cateer resource center It is 
n*-i psv^y for you to develop those support 
(yt ( Nigrum <;u^stion,4 Their purpose is to 
ru;iKf» wo\t* specific. \hf* qeneral interest in 
ratr'M irsnijM,*' r-nUM iORO) and its 



client services For example, you might 
ask-- 

• Is thf>re a written statement of objectives 
for the center 9 , 

• Are Clients finding in the center the types 
of resource materials they rieeri in explor- 
inq potential careers' 7 , ami 

• Is the center's schedule conolntpd with 
clients" needs 9 

In short, ynu need to develop n coniprrhen- 
srve set of questions like those lm tnu:h ot 
the ' oad key questions included in your 
<v. nation plan This is the hrst step in 
developing or selecting instrument items 

? Prepare or select the items for the instru- 
ments you will use in gathering evafuative 
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data for your career guidance program* 

Whether constructing or selecting items, 
you should ensure that each is directed 
toward answering one or more selected cri- 



terion questions. For example, the following 
items would be appropriate for the question 
Is there a written statement of objectives for 
the center^ 



Are the objectives for the CRC- Yes Somewhat No 

• available to clients, 

• measurable. 

• clearly written, and 

• understood by clients? 

The question "Are clients finding in the center the types of resource materials they need in exploring 
potential careers?" could be answered with the following items. 

How adequate are the center's resource materials in your area{s) of career interest? 

Very 

Printed Materials Adequate Adequate Inadequate 

• Books 

• Directories 

• Brochures 

Audiovisual Materials 

• Films 

• Filmstrips 

• Tapes 

Manipulative Materials 

• Computers 

• Games 

• Kits 

The third question. Is the center s schedule correlated with clients* needs?, could be answered with a 
simple rating scale, 

How would you rate the schedule of hours maintained by the CRC? 

Always open when you need it Never open when you need it 

Usually open when you need it . . _ No opinion 

Seldom open when you need it 

In summary, developing your evaluation instruments should begin with the areas of concern you 
decide tc include in the evaluation plan and conclude with specific items to be included in the 
evaluation mstrument(s). The relationship between areas of concern, key and criterion questions, 
items, and instrumentation for gathering data are depicted on the next page, 
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E 



KEY QUESTIONS 








CRITERION 
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EVALUATION 
INSTRUMENTS) 



Assemble the evaluation Instruments, pilot* 
test them, ano make necessary changes* 

After you have selected or developed the 
needed items, they should be grouped 
according to their area of concern, the types 
of rating scale they use, and the information 
sources(s) to which they will be directed. 
Prior to the actual evaluation, pilot-test all 
items using small groups of persons who are 



similar in characteristics to those you will 
use as information sources, such as clients* 
employers* and parents. You can then make 
the necessary revisions and refinements in 
the items. Review thecomments obtained as 
well as the results of the trial run and make 
the recommended changes. If you think it 
necessary* another trial run should be made. 



Collecting and Analyzing the Data 



, - ■ ** .- , - in'! 
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Apply the Mlected evaluation measures to asaeas 
ComptteflCy § the«ttelnfn#ntof progr^rn process and performance 

i : -Y- .- objectives* ' 



The next step is to administer the evaluation 
instruments Activities for your evaluation team at 
this stage of the evaluation include the following: 

1 Develop lists of persons to be surveyed as 
you gather evaluative data and information* 

For example, these lists would include cur- 
rent and former clients if one of your areas 
of concern is client services, In all instances, 
listings should include only those informa- 
tion sources that are in the best position to 
respond to your inquiries and provide the 
information needed to answer your criterion 
(and ultimately key) questionsforeach area 
of concern included in the evaluation. 

2 Train tho*e who will be collecting data* To 

maintatn consistency as you Interview and 
administer instruments, provide some inser- 
vice training for those doing these tasks. All 
clients, for example, should receive the 
ame verbal instructions for completing any 
questionnaire you might use. Likewise, 
interviews should be consistent and proper 
techniques for questioning and recording 
responses must be observed. 

3 Mall survey Instruments* Many former cli- 
ents, employers, and the like will have to be 
contacted by mail Develop a suitable cover 
letter that explains the purpose of the en- 
closed survey instrument and encourage 
the prompt return and careful completion of 
the instrument Also, consider using a cod- 
ing mechanism to readily follow up on those 
in the survey sample who do not respond 
promptly 

4 Administer survey instruments* You should 
have few if any problems administering 
instruments to current clients and others on 
your lists of rnformation sources who are 
roadily available Remember the telephone 
interview is an excellent technique to survey 
busy people and others who are not readily 
accessible in person. Also, set a target date 
for completing this process and then strive 
to merl it Keep a record of the planned 



group administrations and individual inter- 
views as they are completed This will help 
ensure that the target date will be met. 

5. Follow up or* nonrcspondents* Keep a record 
of mailed instruments as they are returned 
A follow-up card or letter should be devel- 
oped and sent to persons who have not 
responded within a reasonable length of 
time, Telephone those who do not respond 
to the follow-up card or letter and solicit 
their cooperation in completing and return- 
ing the survey instrument. 

In summary, you should develop well-coordinated 
plans for data collection. Remember, you want to 
get maximum information with minimum effort 
and still maintain positive relations with clients, 
employers, and others whom you will be asking to 
supply you with information. 

Once you have collected adequate information 
and data (as much as you can reasonably expect 
to obtain within the limitations imposed by time 
and money), tabulate and code that which you 
have collected, Depending on the instruments 
ou use and the », umber of respondents, you can 
choose totally responses by hand or by mechan- 
ical means. Of course, responses to open-ended 
questions and the like must be analyzed and tal- 
lied individually. If there are less than 50 persons 
responding and the instrument is short, you may 
find it quicker and less expensive to hand tally 
even objective responses. However, large num- 
bers of responses to yes-no or multiple^choice 
survey items usually warrant a mechanical tally 

When preparing the evaluation report you should 
do the following 

1 State your purpose for conducting the 
evaluation 

2 Describe how you collected the evaluation 
data and information, 

3 State your findings in a ctear and concise 
manner and relate them to the areas of con- 
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corn and thuhroad key questions addressed 
by the evaluation activities, 

4. Make use of simple techniques for summa- 
rizing data and information, Raw data such 
as the number of "agrees" and "disagrees* 1 
show a partial picture, Present them also as 
percentages, ratios, or average responses. It 
makes them more meaningful and easier to 
understand. For example /70 percent agree 
and 30 percent disagree/ 1 "7 out of 10 
agree," or a bar graph showing the rela- 
tionship between these 2 repsonses have 
more meaning toreadors than simply report- 
ing 210 clients agreed and 90 indicated 
disagreement. 

5 Formulate and present in the report conclu- 
sions, recommendations, and suggested 



actions. Conclusions are summary state- 
ments of the report findings that serve to 
answer the key questions for the evaluation. 
For example, your data might lead to a con- 
clusion such as. "30 percent of the clients 
indicated that they have not used the career 
resource center/* A corresponding recom- 
mendation might be to"encourageall clients 
to use the services of the CRC," Recom- 
mendations are broad statements indicating 
how you and your co-workers should re- 
spond to the conclusion statement, Sug- 
gested actions are specific statements or 
"action steps*' that might be taken in imple- 
menting the recommendation. An example 
of how you might organize conclusions, 
recommendations, and suggested actions 
follows: 



AREA OF CONCERN-Career Resource Center 



CONCLUSION 



RECOMMENOATION 



SUGQESTEO ACTIONS 



Thirty percent of the clients 
indicate they have not used 
the CRC 



Encourage all clients to use 
the resources of the CRC, 



• Develop and distribute to clients 
a brochure describing the ser- 
vices of the CRC. 



• Appoint a committee to conduct 
a survey to determine why clients 
are not using the CRC. 

• Develop a slide-tape presenta- 
tion of the CRC and its services 
to be used in client orientation. 



6 Review the report with the evaluation team. 

Once the initial draft of your report is com- 
pleted, it should be reviewed section by sec- 
tion with your evaluation team Each member 
should have an opportunity to peruse it and 
propose and consider re commendations for 
improving it The value of the entire evalua- 
tion activity can be lost if your final report is 



anything less than 100 percent accurate or 
shows evidence of carelessness or thought- 
less expression, Be able to substantiate, if 
necessary, all statements in the report as 
well as make sure that each is clear to all 
intended readers. Thus, this final review by 
the evaluation team is critical It should not 
be taken lightly 
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Evaluating the Evaluation 



> Evaluate the proceat used in the formative evalua- 

COHratttttOY 7 " '■ 1 tlon W the program to make neceaaary alterations in 



As indicated earlier evaluation data are planning 
data tor making formative or enroute planning 
decisions as well as making summative orend-of- 
program decisions The reason you evaluate your 
program is to gather data for making plans to 
improve your career guidance program, How- 
ever, your evaluation activities should not stop 
here Also, evaluate the processes used in evalu- 
ating your programs, Evaluate your evaluation, 
On what other basis would you make necessary 
changes in your evaluation process? 



Designing and conducting a full-scale evaluation 
of your evaluation process is a rathei complex 
activity, It involves nearly all the evaluation pro- 
cedures and techniques presented thus far in this 
module, For that reason, it is often a neglected 
activity. In the absence of a full-blown evaluation, 
however, the following questions will help you to 
determine the quality of your evaluation efforts, 



YES NO 

Is the evaluation continuous throughout prog* am development and implementation and 
not |USt a year-end activity? 

Does the evaluation focus on assessing the attainment of stated program objectives and 
clients' reactions to program services and activities? 

Are both formative (process) and summative (product) data collected? 

Are the data collection processes carefully planned and coordinated to obtain useful data 
and avoid duplication of effort? 

Is consideration given to various techniques and instruments for collecting data and is 
selection oased on the data needed, persons surveyed, and local constraints? 

Is consultative assistance used when needed to expedite the development of the evalua- 
tion plan and evaluation instruments and in the analysis and interpretation of the 
obtained data 7 

It is hoped that you will be able to check "yes " for each of the above questions as they relate to your 
career guidance program evaluation activities. 
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Learning Experience 1 

Writing Evaluation Questions 



OVERVIEW 



COMPETENCY 



READING 



INDIVIDUAL 

LEARNING 

OBJECTIVE 



INDIVIDUAL 
ACTIVITY 



INDIVIDUAL 
FEEDBACK 



GROUP 

LEARNING 

OBJECTIVE 



GROUP 
ACTIVITY 



Verify the relationship of the components of the activity to be 
assessed, including statements of goals, client performance 
objectives, end process objectivea. 

Read Competencies 1 and 2 on page 9: 



Write key evaluation questions to be answered In theaMpas- 
ment of client performance objectives for selected prograrn 
components of areas of concern in the evaluatt'o^ .!> 



Develop client performance-oriented Key questions* 



Check your key questions against the criteria provided. 



Write key evaluation questions to be answered in the assess- 
ment of program process objectives for selected areas of con- 
cern in the evaluation. 



Develop program process-oriented key questions. 
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INDIVIDUAL 
ACTIVITY 



Develop client performance-oriented key questions, 



Prior to starting this activity, review the reading for Competencies 1 and 2 on page 9, 

In this activity you will develop a series of key questions that will assist you in focusing you revaluation 
activities on the products of your career guidance program, that is, your clients' knowledge, under- 
standings, decision-making and coping skills, and attitudes that are important to entering and serving 
effectively in productive and rewarding careers. 

First deciae upon and make a listing of the areas of concern in your clients* development that you want 
to include in the evaluation activity. Your list might include areas such as clients'— 

* knowledge of the world of work, 

* ability to select and enter appropriate educational programs. 

* learning skills and values. 

* identity and self-image, and 

* interpersonal relationships. 

You are now ready to develop your key questions for the areas of concern included in your list, 
Remember the purpose of the key questions is to help you focus your evaluation activities, 

Start with the area of concern in your listing that has the highest priority and develop a key question for 
it An example key question for the product area "knowledge of the world of work" might be as follows: 

To what extent are clients knowledgeable of those aspects of the world of work th^t are 
important to making appropriate career development choices? 
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INDIVIDUAL 
FEEDBACK 



Check your k«y questions egstnet the cri(#ri« provided; 



YES NO Examine the following aspects of your key questions: 

1 Do your key questions focus on the product of your career guidance program— your 
clients and their growth and development? 

2 Do they ask a question that, when answered, will contribute to your evaluation of its 
related area of concern? 

— 3. Are they broad enough to accommodate two or more specific criterion questions from 

which items for the evaluation instruments can be developed? 

. _ 4. Are they questions that can be answered with the types of data that willbegatheredin 

the evaluation? 

Answers to all four of the above questions will be "yes" for well-developed key questions. 



AcSSJlTY Oevelop program process-oriented key question* 



Note: The following outline is to be used by the workshop facilitator 



Facilitator's Outline 


Notes 


A. Starting Point 




1 Indicate that participants will be 
working independently and together 
in developing and critiquing key 
evaluation questions that focus on 
selected program processes in their 
career guidance programs. 




2. Organize participants into small work 
groups of three to five persons 


The method of "counting off" by threes, fours, 
or fives work well in forming groups. 



3 Have the participants review the 
material on key questions in the 
reading tor Competencies 1 and 2 
on page 9 



9 

ERIC 



21 



Facilitators Outline 



Notes 



B Discussion of Terms 

1 Discuss with the total group the 
meaning of each of the following 
terms as they relate to designing a 
plan to evaluate a career guidance 
program. 

• Program process areas of con- 
cern 

• Key questions 

2 Draw from the group and discuss a 
tew example program process areas 
of concern and related key ques- 
tions that might be used in evaluat- 
ing a career guidance program. 

Identifying Program Processes 

1. Ask participants to develop in their 
work groups listings of example 
program process areas of concern 
that might be included in a career 
guidance program evaluation plan. 

2. Ask each work group to present to 
the entire group their listings. Award 
"team points" according to the fol- 
lowing schedule: 

• +1 point for each bonafide pro- 
gram process area of concern. 

• +2 points for each bonafide pro- 
gram process area of concern 
that appears on all other group's 
listing, 

• +3 points for each bonafide pro- 
gram process area of concern 
that appears on no other group's 
listing, 

3 Total and announce the scores for 
each work group. 

0 Developing Related Key Questions 

1 Distribute and discuss with partici- 
pants the criteria for evaluating key 
questions. (Handout titled Criteria 
for Assessing Program Process Key 
Questions) 



Prior to conducting the activity you may want 
to prepare definitions for each term and a 
statement of their relationship in the evalua- 
tion process. 



Ground rules for the competition should be 
presented prior to starting the activity (e.g., 
methodsof scoring and facilitator s judgments 
are final). 

Presenting groups should be able to "defend" 
to others the validity of the program process 
areas of concern in their listings. 



Prior to conducting the activity, prepare for 
each participant one copy of the attached list- 
ing of criteria for evaluating key questions. 
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Facilitator s Outline 



Notes 



Select from among the earlier pre- 
sented program process areas ot 
concern that might be included in an 
evaluation plan tor a career guid- 
ance program a listing ot tour to 
eight (one or two trom each small 
group) good examples and present 
them to the participants. 

Develop and discuss with the partic- 
ipants example key questions tor 
the tirst two areas ot concern in the 
foregoing listing. 

Apply the criteria tor evaluating key 
questionstothosedeveloped in step 
3 and discuss with participants the 
application ot the criteria as you 
apply them. 

Ask participants to develop individ- 
ually example key questions tor the 
remaining program process areas ot 
concern presented in step 2~ 



Chalkboard, flip chart, or overhead projector 
may be used as the presentation media. 



E Summary/Feedback 



1 Summarize the purpose ot and the 
relationship between program pro- 
cess areas ot concern and key ques- 
tions in the program evaluation 
process. 

2 Ask each participant to present one 
or more ot their key questions and 
discuss the appropriateness ot each. 
Note whether their key questions 
meet the evaluative criteria tor same. 
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Handout 



Criteria For Assessing Program Process Key Questions 



1. Does the question focus on an area of concern important to delivering services in the career 
guidance program (i.e.. program objectives, client services, administrative services, facilities, pro- 
fessional staff)? 

2. Will the answer to the question contribute to the evaluation of its related area of concern? 

3. Is the question broad enough to accommodate two or more specific criteria from which items for 
evaluatiott instruments can be developed? 

4. Can the question be answered with the types of data that can be gathered in the evaluation? 
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Learning Experience 2 




Preparing the Evaluation Plan 




OVERVIEW 


COMPETENCY 


Select or develop an appropriate evaluation design that 

Inctudcis a cOmDAfisOn standard lor dAtAfmlnlnn If nainfl mad a 

by the client can be attributed to the program. 


READING 


Read Competencies 1 and 2 on page 9. 


INDIVIDUAL 

LEARNING 

OBJECTIVE 


Describe possible applications of common experimental de- 
signs to determine If client gains can be attributed to a particu- 
lar career guidance program activity or service. 


INDIVIDUAL 
ACTIVITY 


Develop descriptions of ways in which experimental designs 
might be used in evaluating your career guidance program. 


INDIVIDUAL 
FEEDBACK 


Check your suggested applications against a list of criteria. 


GROUP 

LEARNING 

OBJECTIVE 


Design an evaluation plan that will demonstrate the impact 
that a selected program component has on intended gains 
made by clients. 


GROUP 
ACTIVITY 


Design a portion of an evaluation plan. 



INDIVIDUAL 
ACTIVITY 



Develop description* of waye tn which experimental dealgna 
might be uied In evaluating your career guidance program, 



in this activity, develop for each of four different experimental designs an example or hypothetical 
application of how it could be used in the process of evaluating your career guidance program. In each 
instance, your purpose for including an experimental investigation in your evaluation plan will be based 
on your need to demonstrate empirically that certain gains made by clients are indeed a direct result of 
your program. 

Information from Evaluation and Educational Decision-Making (Development Associates 1975) is 
included with this learning experience. It describes in detail four experimental designs: 

• Posttest comparison with matched groups 

• Pretest/posttest control group 

• Norm reference design 

• Criterion reference design 

After completing the reading leferenced for this activity, develop descriptions of how you could use 
each of the above four experimental designs in evaluating your career guidance program. 
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Experimental Designs 



Presented below are several designs that experience indicates will be relevant for most career guidance 
urograms As suggested earlier it is unlikely that any one design will be appropriate for all questions, 
and therefore, more than one of those discussed may be used in a single program, There are also more 
complex designs, of course, which are appiopriate in some situations. More detail regarding the 
designs below and several other alternatives may be found in texts on evaluation and research design/ 



Poattest Comparison with Matched Groups 

This research design requires that a group of potential participants (at least twice as large as the 
desired number) be pretested, that those tested be paired in terms of pretest measures, and that one 
member cf each pair be randomly assigned to the participant group and the other to the comparison 
group The postiest then measures the same skills as the pretest. This design provides an accurate 
estimate of the impact of the program (oractivities) on participants. However, it is a somewhat difficult 
design to ust? in a school setting. 



Pretest/Poaltest Control Group 

This research design requires (1) random assignment, prior to treatment, either of individuals to 
participant and comparison groups or of program activities to some existing but essentially similar 
groups (e.g. classrooms) and not toothers; and (2) pretesting andposttesting of the participants and 
comparison groups. This design can provide an accurate assessment of the Impact of the program on 
participants only if the pretest differences between participant and comparison groups are due to 
chance. Although it may be difficult to meet the requirements of this design on a large-scale basis (eg* 
throughout the program or the scnool district), it may be quite feasible in many situations. 



Norm Reference Design 

This design requires use of standardized tests with pretest and posttest norms and careful adherence to 
the testing procedures followed by the test publisher in obtaining normative data. It does not require 
local comparison groups. It is particularly important to note that testing dates should correspond to 
those used in norming the test (eg. October and May), and that the norm group should be as similar as 
possible tothe participants. This design may provide a quantitatively supported estimate uf the impact 
of ihe program on participants Where standardized tests are available and proper testing procedures 
followed, this design can be implemented at considerably less cost and effort than a comparison group 
design 



Criterion Reference Design 

This design is most appropriate for program objectives dealing with levels of performance (e.g.. the 
number of materials developed, the extent to which teachers use a media center the number of 
students placed in employment, or the number of courses add'^d to the school curriculum) or verifiable 
changes in the school district (e g. a school policy favoring career education, an agreement with the 
employment service, etc.). It provides an estimate of the extent to which a predetermined criterion has 
been achieved. It requires a careful specification of the criterion, and valid and reliable pre- and 
postmeasjres* but may not require a comparison group. Although useful in measuring how close 



*Of pAfticiiliif mliMPStto Ihr manager may ho DonniriT CaropnMlanri JuhanO Stanley EnpctimentatandQuaiiiExfMrinwntai 
0**%t*tns tot tinware [Chicago R^nri McNally 19631 

D Hotst K TaNm«i(1g^ ,indC Wood 4 Ptt\c ticatCiUidotn Mfatsutmg Ptnirct impa< tPnStiritoftt Achwvfintfint (Washington DC 
Us DfprfflfrioM t\\ Health Edurahon anriWHla^ OMmrnl £riiic<iNor> 1975) 
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performance has como to the criterion* unless it is combined with one of the previously cited compari- 
son group designs, it does not establish that the performance was caused by the program 



Survey Design 

This design providesa description or a quantitatively supported estimate of various relevant character- 
istics of a group (e g., knowledge* attitudes* social characteristics) at a particular point or several points 
in time It requires carefully developed questionnaires that address these areas of interest, Although it 
does not permit drawing the conclusion that program activities were responsible for the survey results, 
it may be quite useful in program evaluation. For example* it may indicate that, for whatever reason* no 
further effort in a particular area is necessary or that much remains to be done, 



Summary 

Whatever design is selected* it is important that information on program activities be related to 
estimates of program impact. For example, even in the absence of a group of students who have not 
been exposed in some manner to the program (i.e., no control group), it is generally possible to identify 
the level or amount of program exposure for each student or class* and treat those with the lowest 
exposure level as a comparison group relative to those with the higher exposure level. 

Comparisons between the gains made by the high group and those made by the low group would then 
provide the basic data needed for assessing whether or not the program had any impact on the outcome 
measures, providing that there are meaningful differences between the activitiesengaged in by the two 
groups. If the activities are quantifiable on a fairly fine scale* such as number of hours, the high group 
might consist of the 25 percent of the students who had the greatest number of hours and the low group 
might consist of the 25 percent of the students who had the fewest hours of exposure. If the differences 
are in fact meaningful, the design would lend itself to analysis to determine the strength of the 
relationship between program activities and outcomes. 
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FEEDBACK 



Check to determine it your proposed use of each of the tour experimental designs is consistent with its 
intended use. 

• Your proposed use of each experimental design should be consistent with the expressed purpose 
of the designs. 

• The numbers and types of clients indicated in your descriptions should be consistent with those 
called for in each of the designs, 

* Your proposed treatment(s) tor clients in your program should be consistent with the treatments 
mdicated in each of the designs, 

* The timing oi the treatments indicated in your treatments should be consistent with the timing 
indicated in each of the designs. 



GROUP 
ACTIVITY 



Note: The following outline is to be used by the workshop facilitator. 



Facilitator's Outline 


Notes 


A Introduce Activity 




1 Indicate to participants that they will 
be working in small groups and 
designing that portion of an evalua- 
tior plan that will provide for com- 
parisons for determining if gains 
made by clients can be attributed to 
the program. 




2 Organize participants into small 
groups 




3 Have participants review the infor- 
mation on experimental designs pre- 
sented in the Individual Activity, 
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Facilitator s Outline 



Noes 



B Develop the Experimental Component 
of an Evaluation Plan 

1 Explain that an evaluation plan must 
includeoneormoreof thefollowing 
experimental designs whenever you 
desire to make comparisons that will 
help determine if client gains are a 
result of somespecific program activ- 
ity or service: 

• Posttest comparison with 
matched groups 

• Pretest/ post test control group 

• Norm reference design 

• Criterion reference design 

2 Explain also that the design for the 
experimental component should in- 
dicate which of the above four ex- 
perimental designs you are using 
and include descriptions of the 
folfowing; 



Hypothesis or hunches to be 
tested or questions to be 
answered 

Clients to be included in the 
investigation 

Types of data to be gathered 
Instruments and timetablesto be 
used in gatheitng data 
Manner in which the data ob- 
tained will be analyzed to test 
hypotheses, validate hunches, or 
answer the initial questions 



3 Have the participants, in their small 
groups, generate a description of an 
experimental component for an eval- 
uation plan to determine if some 
selected client gain(s) can be attrib- 
uted to some particular component 
of the career guidance program(s) 
with which the group is associated. 

C Report and Feedback 

1 Bring the participants together and 
have each small group present its 
experimental design. 



Allow 25 minutes for groups to complete the 
above activity. 
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Facilitator's Outline 


Notes 


2 Critique each design as it is pre- 


The basis for the critique should be adequate 


sented and point out its strengths 


application of the selected experimental design 


and any aspects that need 


as described in each of the points listed in B-2: 






• Hypotheses 




• Clients 




• Instruments and timing 




• Analysis 
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Lenrning Experience 3 

Selecting Evaluation Strategies and Instruments 



OVERVIEW 



COMPETENCIES Select criteria tor evaluation procedures and instruments, and 

adapt, idopt or create measures of program process and 

' V'- ■'' ' '^erlprrw^ 




. iMiQci criiflrta for evaluation ifyHrumentB; a no seteci. soapt. or 
create appropriate sommative evaluation Instruments and 


READING 


ftead Competencies 3 and 4 on paga 11 j : . 


INDIVIDUAL 

LEARNING 

OBJECTIVE 


Write orlterlon^ueftttons 3nd related Item* for an inttrumttnt 
that will be useful in assessing the attainment of program . 
process objectives. .'■ : y'.-' '■' ; , ' ; :-..-. : ;'-'V; , - : ' 1 '■"v.: : -; 


INDIVIDUAL 
ACTIVITY 


Develop Items useful in assuring the attainment 61 program 
prdcws obiecftyes, *■ : ; > . 


INDIVIDUAL 
FEEDBACK 


' : .iv . ' ■"" /- ' -* ' * - v h ■ "' . ' - i - *„ ■ ' ■■ . ■ 

Check your deveJopad items against a set of .generalized 
criteria. ; ' •,. ' „■.:". •. ."Vv. 


GROUP 

LEARNING 

OBJECTIVE 


Adapt existing n measurement instruments for osein assessing 
the attainment of the program process objectives lor your 
, career guidance program. 


GROUP 
ACTIVITY 


Adapt a given Instrument for use in evaluating a career guid- 
ance program. 
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Prior to starting this activity, review the reading for Competencies 3 and 4 on page 11. In this activity, 
develop items that could be included in the various types of instruments that might be used to assess 
the attainment of the program process objectives for your career guidance program. This activity is 
intended to be as practical as possible. Therefore* as you develop the requested items think of how they 
might be used in evaluating the program in your agency, 

First, select one process-oriented area of concern for your career guidance program. Examples are 
indicated in the reading. Other examples include the folfowing: 

• Professional staff 

• Facilities 

• Program objectives 

• Client services 

• Administrative services 

Next, develop one or more key questions and several related criterion questions for the area of concern 
you have selected. Both types of questions are defined and discussed (complete with examples) in the 
reading. Other examples of one program process-oriented key question and related criterion questions 
follow: 



Key Question 

Is the career guidance staff adequate to develop 
and deliver a viable career guidance program? 



Criterion Questions 

1. What is the client staff ratioforthecareerguid- 
ance program? 

2. (s adequate secretarial support available for 
guidance staff? 

3. What types of professional preparation and 
certification do professional staff have? 



Once you have your listings of key and criterion questions developed, determine the best source for the 
information needed to answer these questions. The usual sources are presented and discussed in the 
reading and include the following: 

• Current and former clients 

• Career guidance staff 

• Agency administrators 

• Clients' employers 
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Select one or moro information sources for each of your listed criterion questions and develop items 
that will be useful in obtaining from these sources the data and information needed to answer each of 
your criterion questions and ultimately your key question(s) for the area of concern you selected. 




Chocfc your dt wtop o d ttm$ io*)n<t < Ml of ganerallzad 



Ye* No 

— - 1 Does the item relate to the content of the criterion question? 

— . 2 Is it valid— actually measuring what it is intended to measure? 
3 Is it reliable-providing accurate measures? 

4. Is the Hem relevant to the evaluation task at hand? 

5 Is it cost-effective-providing needed information while requiring no morethan reason- 
able amounts of time and other resources? 

6 1s the item trouble free—providing needed information while not risking violation of 

privacy laws, excessive complaints, and the like? 

Answers to all six of the above criteria will be "yes'* for items that you would want to use in evaluating 
your career guidance program. 




Note: The following outline is to be used by the workshop facilitator. 



Facilitator s Outline 


Notes 


A Starting Point 




1 Indicate that the participants will be 
involved in independent and group 
activities while examining and modi- 
fying a published evaluation instru- 
ment to make it better fit their data 
and information needsin the evalua- 
tion of their agency's career guid- 
ance program. 




2 Divide the participants into small 
feedback groups of three or four 
persons 
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Facilitator 4 ! Outline 



Notes 



3 Have the participants review the 
material on instrument development 
in the reading for Competency 3 on 
page 11 

B Review and Modify the Evaluation 
Instrument 



1 Indicate that the participants will 
individually complete the following 
tasks; 

a Review each item on the evalua- 
tion Instrument. 

b Rate each item as to its current 
state of usefulness to an evalua- 
tion of their career guidance 
program. 

Thefollowing rating scale should 
be used in this activity: 

• 3—very useful in its present 
form 

• 2-useful as is but could be 
made more so 

• 1-needsmajorrevisiontobe 
made useful 

• O-cannotbemadeintoause- 
ful item. 

c Revise 15 or more items that 
received ratings of 1 and 2 so 
that each would merit a rating of 

3. 

2. Distribute the abbreviated copy of 
the instrument entitled Indirect Ser- 
vices Coordinated by Guidance Staff 
that appears on the following pages. 

3. Ask participants to take from 30 to 
40 minutes to complete the above 
activities, 



Prior to conducting the activity, you will need 
to duplicate one copy of this instrument (or 
another one of your choice) for each participant. 



C Review and Feedback 



1 Indicate that participants will be 
working in their small feedback 
groups to review and evaluate the 
item ratings and suggested modifi- 
cations of the other participants in 
the group. 
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Facilitator's Outline 



Notes 



2 Ask participants to report in their 
groups, item by item, the action they 
have taken (retain as is, modify as 
follows, or reject outright) and the 
reason tor their action. 

3 Encourage group members to try to 
reach some consensus as to the 
proper treatment of each item as the 
instrument is adapted tor use in 
evaluating their agency'^ program. 

4 Refer participants to germane 
sections of the reading in those 
instances where feedback group 
members cannot reach consensus 
on one or more items. 
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Handout 



Indirect Services Coordinated by Guidance Program Staff 



The rating scale is as follows: 1-provision met exceptionally well; 2-provision met adequately; 
3- adequate but improvement needed; N-not applicable; ?~not enough information to make 
judgement. 

1* Student Information 

1 2 3 N ? A. A guidance staff member has the responsibility for developing and coordinating 
the student information service. 

t 2 3 N ? B. Each student's record is separate from the student's school permanent record. 

1 2 3 N ? C Information useful to the student, parent, counselor* and teacher is systematically 
collected and recorded on each student's guidance record including such infor- 
mation as school marks, test performances, school activities* expressed interests, 
vocational preferences, continuing education preferences, special talents and 
achievements, and wo/k experiences. 

1 2 3 N ? D. Each student participates in contributing and maintaining information in his or 

her guidance record* 

1 2 3 N ? E> The information in each student's guidance record is available for interpretation 
to students, their parents, and school staff. 

1 2 3 N ? F. Student evaluative information is given only to the students, their parents* those 
working professionally with the students* and those for whom approval Is given by 
the students. 

1 2 3 N ? G + Student guidance records are easily accessible to the school staff. 

1 2 3 N ? H. Observations that make test performance or other data more meaningful are 

recorded on the guidance records. 

\r Organized and coordinated procedures and techniques are used to ensure that all 
students are identified who- 

1 2 3 N ? (1) have special talents, abilities* achievements* and interests; 

1 2 3 N ? (2) have special learning needs or handicaps; 

1 2 3 N ? (3) are potential dropouts; 

1 2 3 N ? (4) seem to be making inappropriate educational and vocational plans; 

1 2 3 N ? (5) have made a significant drop in school performance: 

1 2 3 N ? (6) have special health needs or physical handicaps; 
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1 2 3 N *> (7) have negative self-concepts; 

1 2 3 N ? (6) have negative attitudes toward other individuals and society; 

1 2 3 N ? (9) need financial assistance for continuing education beyond high school; 

1 2 3 N ? (10) should be referred to other pupil services personnel or guidance resources, 

J. A coordinator of the testing program, in cooperation with the building administra- 
tor organizes and coordinates- 

1 2 3 N ? (1) the securing, distributing, scoring, and storing of test materials; 

1 2 3 N ? (2) the planning of testing schedules, training of test examiners, orienting of 

staff and students, identifying of facilities, and the carrying out of other 
activities related to proper test administration; 

1 2 3 N *> (3) the developing of test interpretation materials, procedures, and reports 

that make test results meaningful to students, parents, teachers, and 
administrators, 

1 2 3 N ? K. At lea.U one trained proctor is provided for each 30 students tested, 

1 2 3 N ? L Student test data are reported in types of scores that will facilitate accurate 
interpretation of the students performance. 

1 2 3 N ? M. School ability tests are administered at approximately the same time as the 
achievement tests and have norms comparable to the achievement test, 

1 2 3 N ? N. Vocational aptitude and interest inventories are administered in either grade nine 
or ten depending upon the organization of group guidance instruction and voca- 
tional education curriculum, 

1 2 3 N ? O, Appropriate tests are used to assist in the identification or screening of students 
with needs for remedial reading, study skills instruction, remedial arithmetic, and 
other special programs, 

1 2 3 N ? P. Provisions are made for students to participate in state, and national testing 
programs which facilitate the accomplishment of guidance objectives. 

Q. Ability and achievement test results are organized for the purpose of— 

1 2 3 N *> (1) describing the means and variabilities of the student population; 

1 2 3 N ? (2) identifying possible needs for special education, occupational work expe- 
riences, remedial reading, or other curricular programs; 

1 2 3 N ? (3) describing a student's test performance in relation to those in his own class 

and school 

1 2 3 N *> R Expectancy tables are developed to describe potential student performances in 
high school subjects and college. 
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S Follow-up studies are conducted on graduates and dropouts at various intervals 
lor the purpose of collecting information about their- 

1 2 3 N * (1) employment; 

1 2 3 N o (2) post-high school education; 

1 2 3 N ? (3) opinions on the adequacy ot the school curriculum* guidance programs* and 

student activities; 

1 2 3 N ? (4) recommendations tor improvement of the school curriculum* guidance pro- 
gram, and student activities, 

1 2 3 N ? T. The guidance program is evaluated periodically to determine guidance accom* 
plishments and needs tor further development. 

1 2 3 N ? U Students are surveyed periodically to determine their opinions and perceptions of 
the school curriculum, guidance program, and student activities. 

2. GuManc* Information 

1 2 3 N ? A. A guidance staff member has the responsibly tor developing and coordinating 
the guidance information service. 

B. The following types of informational materials are available to students and 
teachers: 

1 2 3 N ? (1) School programsof study (e,g,.coursedescriptions, graduation requirements) 
1 2 3 N ? (2) College bulletins and catalogs 

1 2 3 N ? (3) Directories of educational opportunities (e.g., universities, colleges, junior 

and technical colleges, specialty schools, apprenticeships) 

1 2 3 N ? (4) Military services and draft information materials 

1 2 3 N ? (5) Local* state, and national scholarship and financial aid sources 

1 2 3 N ? (6) Study skills materials 

1 2 3 N * (7) Vocational career information(e,g., Occupational Outlook Handbook* DOT* 

Employment Information Series) 

1 2 3 N o (8) Personal-socio materia- ^ rg„ values, social skills, individual differences* 

interpersonal relation ' 

1 2 3 N *> (9) Current guidance evei.v? Mhe school and community 

1 2 3 N ? C> Guidance information is lotted in the guidance room or counselor s office and 
the library. 

1 2 3 N *> 0 Controlled procedures art* followed to make guidance information easily accessi- 
ble to students and teachers 

1 2 3 N <> E An organized filing system is used to categorize vocational information. 
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•> 
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1 


2 


3 


N 


? 


G 


1 


2 


3 


N 


•> 


H 



information has not changed 

1 2 3 N ? I Information is available on the employment and continuing education of the 
school's graduates. 

1 2 3 N ? J. The guidance office is on the mailing lists of various publishers (e.g. superinten- 
dent of documents, state department of education, state employment office, 
commercial publishers). 

1 2 3 N ? K. Bulletins or newsletters are periodically distributed to students and parents con- 
cerning activities or events in guidance. 

3. Outdance Resources 

1 2 3 N *> A, A guidance staff member is assigned to coordinate the guidance resource service. 

B< Organized provisions ore available for the purpose of assisting students, parents, 
and staff to- 

1 2 3 N ? (1) use referral resources in the school system which might be of help to them, 

and 

1 2 3 N ? (2) use guidance resources in the community that might help them achieve 

guidance objectives. 

1 2 3 N ? C. A written list has been prepared of competent persons in the community and staff 
who are willing to talk with a student or group of students about a specific 
occupation. 

1 2 3 N ? D Communication and cooperative relations with each guidance resource is main- 
tained through periodic contacts. 

1 2 3 N ? E. Available community guidance resources are being used to help achieve guid- 
ance objectives (e.g., employment service, military services, vocational rehabilita- 
tion service, vocational consultants, professional societies). 

4. Student Placement 

1 2 3 N ? A. A guidance staff member has the responsibility for developing and coordinating 
the student placement service. 

1 2 3 N *> 6 Applications and admissions materials are processed to facilitate student entrance 
into colleges and other educational programs. 

t 2 3 N *> C Students are assisted to visit post-high school educational institutions. 

1 2 3 N *> D Student applications for scholarships and financial aids are processed 
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1 2 3 N ? e 



Written records are kept of each transaction that the school performs in process* 
mg student applications. 



ERIC 



1 2 3 N ? F, Procedures have been established to assist students to obtain employment. 

t 2 3 N *> G Cooperative relationships have been established with the local state employment 
office, 

1 2 3 N ? H Vocational exploratory experiences available to students have been identified. 

t 2 3 N ? I. Any decisions needed on the placement of students into curricular programs or 
groupsare made by the administrator taking into consideration the recommenda* 
tions of teachers and counselor the desires of the student and his or her parents, 
and the needs of the student. 

1 2 3 N ? J. Grouping and other curricular placements of studerts are flexible enough to allow 
for changes and adjustments when desirable. 
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Learning Experience 4 

Collecting and Analyzing the Data 



OVERVIEW 



COMPETENCIES 


Administer the evaluation instrument* according to i 
ation design, «nd analyze and 4aplay the evaioetic 
usable term ( ^-.--.'c i;V^^^: ? --->^="'-\-^'. ^> 


iheevatu* 
m data in ^y: 

Vi'' '■ " '■■ '■' ' ■' '''1 


READING 


Apply th« stifled «nmtt1onmeiturt« to Htettti 
mtnt&tpfoorimprO(W»«rKipertOfmim!tobiiCtty 

^Be&tim^ 


fie attain* 

y r ; '■ -r, 

{.'■: v ---i''' , ',."'''-i" l .'' 


INDIVIDUAL 
LEARNING ^ 
OBJECTIVE 

• i, ' '.'r ► 'V,' J V ,' J| ' 'V 1 


<Vv:-^-'^^ yf : ys^. 

Analyze and display evaluatl^ 


;■■ \ '' l /,<;. 1 ',tri 

r y\yy^y-\ 


' ' - " ■ ' 1 , - r - ,h ^' h , 'S h / V L 

INDIVIDUAL 
ACTIVITY 


' Anatyie. •ummafizt, and dWptay » »et of evafual^ 

: ,'; : v.-..;; ^^i. ;■:"".;>'-';: i-.--^''') C' y^C : iV'yyi 


,' •"I;; 1 ;';- \y i 
yyy'?}r' : 

data, --v;V r v! 

; ! - J ..t/ '- ■=■■ 1 : ► ■' ^ 

* ^ " ' . '■ :■■•/ . 1 


INDIVIDUAL 
FEEDEACK 


r '■ ' ■'. "■' .(■'"' ' <'\-y*y'\ V : ■ '■■^v 1 -^' v ' 

; fluWe!in^§:;.;^';^'; ;:.,:V^ ; -:-- 7^; 

.. ■ ■.» '- , j, ' ' - ' ....".>. - ^ ; '■> ■'■ .' ■ >'.'. ".' ^. ■;■ * .., . 


, L , ■ j ■ ■. ". t * :■ ; m i 

ti^ or 

.: , .."v.v*" i „.-..' ' ,. 


GROUP 

LEARNING 

OBJECTIVE 


■ '■ ■ ■ 1. '■■ ' :*'.,.' ■' V'--. .■ / '* ' ■■" ' : "■■ " J ■ - ; 

"' ' "■■ "' ■ . V' '■■ V '.„ : "'-' ^ 'v 

Adminiater evaluatlott instrument* according to • 
Pten. f .'. .' • •• :.r ^', ; '- 


valuation 

' ' * ■ ' ^ 


GROUP 
ACTIVITY 


Participate in a role^playtno iftuati^n In which yeu * 
evaluation inatrumanta and crUiQwe tha performance 
as they pla/ similar ro<ea. 


dmlnister 
■of others 
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Prior to starting this activity, review the reading for Competencies 5 and 6 on pages 13-14. In this 
activity you will analyze a set of tabulated responses toa survey instrument designed to gather datafor 
evaluating a career guidance program. The data you will be working with appears in Community 
Reaction Survey below. 

Assume that these data were gathered as a part of an evaluation activity for your career guidance 
program. The respondents were a random sample (N=60) of adult residents from your local community. 
You desire to summarize anddisplay these data so as to communicate to the administration, the board 
of control, and the community at large the findings of your survey of community reaction to your career 
development program. 

You are now ready to summarize and display your findings. 



Community Reaction Survey 

Career Development Program Date 

1. In general, I am aware of the Career Development Program. 

_40_ Yes J6_ No 4 Unsure 

2. The major objectives of the program, as you see them, are as follows: 

(please rank with 1 being the highest priority objective and 6 being the lowest priority objective): 







1 


2 


3 


4 


5 


6 


7 


a. 


Meet national labor needs 








10 


40 


10 




b 


Develop specific student job skills before 
graduation 


2 


2 


22 


24 


10 






c. 


Integrate vocational and academic education 


20 


32 


e 










d. 


Help students select an occupation before 
graduation 


22 


14 


e 


12 


4 






e 


Increase the use of community resources 


6 


e 


10 


4 


2 


26 


4 


f. 


Create an awareness of the world of work 


6 


4 


12 


10 


2 


16 


10 


9 


Impossible to determine 


2 








2 


6 


46 


h 


Other 


2 










2 





e 
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In your opinion, are parents supportive of this type of program? 



20 Much _37_ SOmo 3 Little 



No Support 
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Are you supportive of this program? 



35_ Much _23_ Some 



2 Little 



No Support 



General comments: 



5. Up to now my involvement in the career development program has been: (check all that apply) 
JJ3 a. a parent 

J£_ b. a community employer 
3 c. a class resource speaker 
8 d, a school-community tour host 

1 e. other 
27 f. none 

6. I would be willing to be involved in the career development program in the following ways (check all 
that apply)- 

10 a. a class resource speaker 
J2_ b. a school-community tour host 
18 c> a personal interview subject 

2 d. other 
48 u e. not sure 

7. I am answering this survey as: 
41 a a Parent 

J0_ b. a business representative 

7 c a civic organization member 
_1 d. an educator 

1 other 



THANK YOU 
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Program Staff: I would like to assist in the following way (check all that apply): 

Resource Speaker Field Trip Sites Personal Interview Subject 

NAME 

ADDRESS 

CITY STATE ZIP 

TELEPHONE OCCUPATIONAL AREA 
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Guidelines For Summarizing and Displaying Evaluative Data 

1 Target the information to appropriate levels of planning and administration- Administrators need 
different types ol information than, tor example, clients and parents do, 

2. Present your information in a clear and concise manner that Is appropriate tor the audience who will 
read and use it 

3. Provide general information about the evaluation that will be helptul in interpreting and understand- 
ing the findings you present (Le, t purposes, procedures* and the like), 

4. Assist readers in making comparisons with the data you present when corr oarisons are appropriate 
(i.e.. client gains in summative evaluations). 

5. Provide conclusions, recommendations, and suggested actions when appropriate {I.e., program 
processes in formative evaluations), 

6. Use appropriate statistical techniques in summarizing and reporting data, 




Hot*: The following outline is to be used by the workshop facilitator 



Facilitator • Outline 



Notes 



A Starting Point 

1 Indicate that participants will be 
working in a simulation-type activ- 
ity, They will be in small groups 
planning for and playing the role of 
one who has been assigned the 
responsibility for administering an 
evaluation instrument and critiquing 
others in their group who havea sim- 
ilar responsibility. For consistency 
the participants will play themselves 
as they might be called upon to 
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Facilitator's Outline 



Notes 



administer an evaluation device in 
the evaluation of their career guid- 
ance program. 

2 Divide the participants into small 
role-playing groups of no more than 
four participants. 

3. Have the participants complete the 
reading for Competencies 5 and 6 
on pages 13*14, 



B Role Play 



1. Indicate that each group member 
will prepare and act out one of the 
following roles in applying evaluation 
devices in the process of assessing 
the attainment of program objectives: 

• Administeringastandardizedtest 
to a group of clients. 

• Administering a standardized 
instrument to an individual client. 

• Conducting a personal interview 
in surveying community 
members. 

• Conducting atelephone interview 
in surveying clients* employers. 

2. Specify that when not playing one of 
the above roles in theirsmall groups* 
participants will be playing the client. 

3. Give participants 10 to 15 minutes to 
identify and prepare for their dual 
roles in each of the 4 situations. 

4. Allow participants 10 to 15 minutes 
to play out each of the 4 roles and 
allow 5 to 10 minutes in between 
each for group members to provide 
feedback to members on their 
presentations. 

C. Summary 

1 Ask for participants* reactions to the 
activity. 

2 Summarize on chalkboard or flip 
chart the important points that 
emerge. 



Prior to the activity* the facilitator will need to 
secure adequate numbers of props (e.g., instru- 
ments, telephones), and make them available 
to each small group. 
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Learning Experience 5 

Evaluating the Evaluation 



OVERVIEW 





Evaluate the process used In the formative evaluation of the „; 
program to make necessary alterations in tl\ls process. 


READING 


Read Competency 7 on page 10. , 


INDIVIDUAL 

LEARNING 

OBJECTIVE 


Write key evaluation questions and re fated criterion questions . . 
to be answered in evaluating #n evaluation process. 


INDIVIDUAL 
ACTIVITY 


fleadah evaluation report and develop evaluation ques^ona.> ^ ; 


INDIVIDUAL 
FEEDBACK 


Compare your evaluation questions with a sat bl sample ques« 
tions presented in the reading and with the program area 
concerns that are listed ■■ ■ 


GROUP 

LEARNING 

OBJECTIVE 


Draft an evaluation plan for assessing an evaluation activity. 


GROUP 
ACTIVITY 


Develop a plan to evaluate the formative evaluation of a career 
guidance program. 
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ACTIVITY 



fteedsn evt(u«tt©n report and develop actuation questions. 



After you have reviewed the reading (or Competency 7 on page 16 and the enclosed evaluation report 
titled "An Overview of Project People in Peoria," develop a set of key evaluation questions and related 
criterion questions that could be used in evaluating the evaluation activities that formed the basis for 
this report, 

Examples of the type of questions you will develop are as follows; 
Area of Concern Key Questions 



Personnel 



Were appropriate personnel in- 
volved in the development and con- 
duct of the evaluation activity? 



Criterion Questions 

Did the career guidance staff have ade- 
quate input in the design, implementa- 
tion, and reporting phases of the 
evaluation? 



Data 



How usk'uI are the data collected 
for this evaluation? 



Was consultative assistance used when 
needed throughout the evaluation? 

Were local administrators, advisory 
committee members, and clients appro- 
priately involved in the evaluation 
activities? 

Were adequate numbers of persons 
sampled in gathering the data for the 
evaluation? Were both formative and 
summative data gathered? 

Were the data gathered relative to the 
purposes and objectives of the 
evaluation? 
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AN OVERVIEW OF PROJECT PEOPLE OF PEORIA: A CAREER 
EDUCATION PROGRAM FOR CHILDREN IN GRADES K-8 



Dr. Chester W. Dugger, Director 
Mr. Charles V. Pryor, Career Counselor 

tor 

School District 150 
Peoria* Illinois 



(Note: These pages are excerpts from a larger report) 
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Dr. Chester W. Dugger 

3202 North Wisconsin Avenue 

Director* Career Awareness and Exploration Program 



Background 

The People of Peoria Career Awareness and Exploration Program (CAE) in District 150 is the result of 
action taken by concerned and informed members of Peoria's business, labor, governmental, and 
educational communities. These people come from varied backgrounds and have diverse points of 
view on many issues, but they share one major concern-How can the schools better prepare children 
to live in today's and tomorrow's world where the only certainty is change? 

Examining present conditions inside and outside the schools, these people noted many thought- 
provoking factors, ror example, that- 

1. many students are dropping out and even graduating from high school without a saleable skill and 
thus cannot obtain employment; 

2. eighty percent of all students who complete high school never obtain a college degree, yet most of 
these students are enrolled in general education and far too few are receiving an adequate 
orientation to vocations and/or skill training; 

3. present employment trends indicate that, increasingly, preparation for existing jobs is requiring 
specialized training, but that general education and even college work does not prepare one for 
job entry; 

4. there is a trend on the part of parents and students alike to question the relevancy of public 
education; 

5. as our technological society becomes more complex, there is less understanding of the commu- 
nity ard its economic bases; 

6. attitudes toward work roles begin to develop even before the child enters school. 

Soon after Career Education was declared a "national priority" by the U.S. Commissioner of Education. 
District 150 became involved. 



Funding and Duration 

A proposal was submitted to the Office of Education, accepted, and was to have been funded in 
November 1971, However, actual funding was delayed until late January 1972. That precipitated a delay 
in securino staff. The director was appointed in March 1972. and the staff was not complete until April 
1972 The $250,000 project is to end in August 1973, 



Scope 

District 150 is integrating a successful, comprehensive career education program into the curriculum in 
all schools, and involving alt certified staff and students in grades kindergarten through eight. 

Number of Certified Staff Number of Children 

875 17,393 
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Intent and Goals 

The stated intent of the project is to install a totiil* articulated, coordinated career education program to 
provide students with- 

1 an understanding ot careers that serve them* the community, and so forth ; 

2 knowledge and positive attitudes toward self; 

3 an internalized value system which includes a valuing ol the student's own career role and the 
roles assumed by others; 

4 increased skill and experience in the rational process of decision making; 
5* an enhanced understanding of the economic system; 

6 opportunities to develop skill awareness; 

7 an understanding ot the relationship between the world ot work and education. 



Rationale 

"Career Education" has no clear definition. This is, perhaps, one of its greatest strengths* Ideas without 
constraining definitions have generated creative plans and implementation processes in school dis- 
tricts throughout the nation. Principals, teachers, and the career education staff have accepted the 
challenge* Starting with the proposal as a foundation, a substantial, effective structure has gradually 
evolved. Several assumptions have served as catalysts throughout the building process. 

1 In final analysis* the teacher 1$ the curriculum. This assumption led the builders to develop a 
program thatis t first and foremost, "teacher" oriented. Unless the people who design and control 
the learning climate begin to do something different, children will not receive an education any 
better or worse than they have been receiving. 

2 People are more important than products and processes. 

3 If significant change is to occur, some process or some thing must be added and/or subtracted 
from the curriculum, 

4 Considering the time limitation (eighteen months), that which is added and/or subtracted must be 
simple and relatively noncontroverslal 

5 Children are role deprived- the quantity and quality of opportunities for children to take an active 
part in the mainstream of society are few and becoming fewer. (Glasser. Schools Without Failure) 

6 Children are aduM deprived-contacts between adults and children are decreasing and adult-child 
dialogue is. for many students, practically nonexistent, Children are not only role deprived, they 
are role-model deprived, (Bronfenbrenner. The Two Worlds of Childhood) 

7 Development of self- awareness, economic awareness, and career awareness, is enhanced when 
child' en are provided with real role models. In the absence of real role models children often turn 
to frctronal role models such as those observed on television Often these models foster false 
impressions of the real world. 

8 Academic education and career education are notdichotomous,--The motto in Oistrict 150 is it is 
not academic education or career education: it is academic education through career edi'jauon. 
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9 "Doing * leads to "understanding" John Dewey has said: 

I hear and 1 forget. 

I see , , , , and I remember. 
I do — and 1 understand. 

10. A creative curriculum can best be generated by creative teachers placed in an atmosphere 
conducive to innovation. 



Curriculum Development 

District 150 has made no attempt to write a new curriculum. This approach has been tried at federal, 
state, and local levels with varying degrees of success. Time and resource limitations made attempting 
such a task unwise. Since no single approach reviewed was, totally acceptable to a committee of 
teachers and administrators, a novel seldom employed and little researched technique was adopted. 

The usual approach is to develop curriculum and then to develop staff to implement the curriculum. 
District t50 has reversed this process. Utilizing easily understood, noncontroversial techniques, staff 
development has led to curriculum development This process might best be described as an individu- 
alized approach to curriculum development. 



Technique 

Two basic techniques have been employed to catalyze the curriculum development process: the 
"person-centered interview" and the t4 organizing-center approach to planning." While space does not 
permit a discussion of the process or rationale, the "person-centered' interview accomplishes the 
following functions: 

1. Provides an atmosphere that encourages teachers to utilize other adults as educational resources, 

2 Provides an atmosphere in which parents and other adults willingly participate. 

3 Establishes a set of rules where adult-child dialogue cannot be avoided. 

4. Establishes a set of rules putting students in "active" rather than "passive" learning roles. 
The "organizing-center" approach to planning— 
1. accomplishes the integration of career education into the regu'ar curriculum, 
2 encourages the utilization of resources outside the classroom-especially human resources, 

3. emphasizes ' concrete 1 * rather than 'abstract" teaching modes* and 

4. encourages "doing" rather than "talking about." 

Both techniques place the teacher and students in a "learning" role. 




Staff Development Events 

A Career Development Committee ol teachers, counselors, and administrators worked to develop goal 
statements and implementation guidelines. These were carefully adhered to as the project developed. 
Implementation proceeded as follows: 

CAE Staff -Held workshops to introduce career education to each school 

-Held community seminar on career education sponsored by the Career Education 
Advisory Council (lay persons) 

—Conducted 2-week workshop todevelop a cadre of teachers tor each building to develop 
building leadership (135 teachers and 33 principals) 

—Developed a teacher handbook giving concrete guidelines tor meeting the established 
goals 

--Established a professor-consultant plan tor formative evaluation and staff support 

-Conducted districtwide implementation institute—a general introduction to the entire 
certified staff 

-Scheduled cadre teacher workshops-one person per building-tobe held at least once 
per month throughout the school year 

-Provided support services to each building cadre 

-Cosponsored an Action Conference on Career Education, providing valuable informa- 
tion and experience to representatives of school districts throughout Illinois and several 
other states 



Responsibilities and Evaluation 

Many projects have utilized a "curriculum development" approach and have enumerated an extensive 
list of objectives The People of Peoria Career Awareness and Exploration Teacher Handbook outlines 
11 tasks for the proiect staff, 6 specific tasks f or principalsand 5 specific tasks tor teachers. These tasks 
are measurable from both an objective and subjective point of view. The success of the overall project 
must be assessed in terms of four primary questions: 

1 Were the tasks accomplished? (objective) 

? To what degree were they accomplished? (subjective) 

3 What impact, it any. did accomplishment of these tasks have regarding the things that are done to 
and with students 7 (objective and subjective) 

4 if, indeed, something different is happening with and tor students, what are the effects on 
students 7 (objective and subjective) 

At the rmd of the 1972-73 school year one can expect to have some data concerning each of these 
questions Progressing from step one through step tour the information will be more difficult to secure 
arul more difficult to interpret. It is obvious, however, that unless step one is accomplished, there will be 
no step two if there rs no step two accomplish ment there will be no step th ree, et cetera. Accountability 
is thr key 
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The evaluation process is a thread running throughout the entire People of Peoria Project Formative 
evaluation provides a feedback loop suggesting modifications as the project progresses Elements of 
this system include the following: 

1 Monthly teacher reports 

2 Professor-consultant reports every other month (11 professors representing 5 universities have 
contracted with District 150 to spend not less than 3 full days per month working in specific 
schools) 

3 Feedback during monthly cadre workshops 

4 Baseline data collected by the summative evaluator via pretest questionnaires 
Summative evaluation will provide data concerning the overall effectiveness of the project. 
Elements of this system include the following: 

1 Teacher questionnaires 

2 Student questionnaires 

3 On-site visitations 

4 Analysis of formative evaluation data 

A great deal of useful baseline data was collected during the November 1972 administration of student 
and teacher questionnaires. For example, the following responses were recorded. 



STUDENTS (GRADES 4*8) 



Question 



We should study about jobs and careers in school 

I am too young to think about my future, 

I like it when we have visitors in our classroom. 

Schools should have more time to discuss how to get a 
job 

How many tj me s did you talk to visitors in your class- 
room last year about their jobs? 



In thn last year, how many times did your class take a 
field trip to learn about jobs or careers? 



Percentage of Responses 



Agree 


Disagree 


No 
Opinion 


No 
Responses 


71.5 


8.9 


17.5 


2.0 


16.4 


72.5 


10.3 


0.8 


76.1 


8.5 


14.3 


0.9 


59.8 


I4.9 


24.7 


0.6 



52% said "none" 

?7% said 2 times or more (of those who had 
classroom visitors) 

42% said "none" 

66% said 2 or more (of those who took field 
trips) 




TEACHERS 



Question 



Career education activities should be treated as an 
additional subject by setting aside specific time and 
using specific instructional materials. 

The "organizing-center" concept allows teachers to 
plan people-oriented activities early and within their 
regular ongoing curriculum structure. 

After the introduction of the program* were you clear 
about its goals? 

Do you feel the goals are worthwhile? 

I n order tofulf ill your role do you think you will have to 
make any changes in your behavior? 

Do you feel there will be benefits for the students? 



Do you think that interviewing people about their jobs 
is worthwhile? 

What is your reaction to the curriculum guide? 

Elementary school is too early to start career aware- 
ness education. 

School dropouts are basically uninterested in academic 
subjects 

Career awareness education should be used to help 
motivate potential dropouts to stay in school, 

Career awareness in elementary schools is the first 
step in planning a successful career 



Percentage of Responses 

No No 
Agree Disagree Opinion Responses 



11 7 79,2 



70.6 



7.8 



6.6 



17.5 



2.4 



4.1 



86.9% said "yes** 

91.2% responded positively 

79.9% indicated some change necessary 

55.8% said "yes" 
38.8% said "maybe" 

88.6% said "yes" 
90.3% said "helpful" 

81.2% disagreed 

70.5% agreed 

84.8% agreed 

65.4% agreed 
12.8% disagreed 



Program Difficulties 

This project has enjoyed tremendous success, but it has labored under a serious handicap. Due to late 
funding the launching of the program was delayed at least 3 months. Staff procurement and orientation 
was completed late in the school year. By the time the summer career education workshop for cadre 
teachers was announced, many teachers already had summer commitments. The career education 
staff had little time to explain the program to teachers. End of the year pressures resulted in teachers 
who were often somewhat less than enthusiastic about any new proposal. The project provided 
stipends for 190 teachers to attend the summer workshop. Only about 135 teachers attended. 

Fora protect this size the career education staff should have been larger. By utilizing consultant monies 
to hire 1 1 professors representing 5 local universities for a period of 9 months, part of this need has been 
met 
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The contractual stipulation that no equipment can be purchased with project funds has been confining 
Requests for hammers, saws, et cetera, should have heen honored. This is especially true with regard to 
special student groups who desperately need "hands-on" kinds of experience. 

Career education has more than a foot in the door in District 150, but innovation must become 
noninnovative if the accomplishments are to have lasting significance. Removal of financial support 
from the project without allowing sufficient installation would be a mistake. The People of Peoria 
Project has served as a catalyst to the development of career education programs in school districts 
throughout the state and nation. Continued support in Peoria will mean continued support to these 
satellite programs as well. 



Evidence of Effect) veneiat 

One need only visit Peoria to answer this question, Evidence is everywhere. Our news media carry 
stories almost weekly. Written teacher reports are on file. Student products are visible in the schools. 
Letters from visitors to District 150 are on file. Requests for consultant help from school districts in and 
outside of Illinois are received weekly. Requests for the People of Peoria Career Awareness and 
Exploration Teacher Handbook number in the hundreds. Student teachers from Bradley University and 
Illinois State University are developing and teaching units utilizing "career education" as the 
"organizing-center." Parents are beginning to initiate contacts to schools asking when they are going 
to be used as resources, Places of business are initiating contacts to the career education office 
volunteering their personnel and facilities as educational resources. The director of instructional 
materials in District 150 reports an increase in the utilization of career oriented instructional materials— 
for example* the librarian at Trewyn Junior High recently complained, "I can't keep up with requests for 
career oriented books." A Career Education "Action" Conference recently held in Peoria and jointly 
sponsored by the state funded ABLE Model Project at Northern Illinois University and District 150 drew 
240 participants from within and without the state of llllnols-among the states represented were 
Oregon, Texas. Michigan, Indiana, Iowa, Kentucky. Nebraska, and Missouri, 

The testimonies that follow tell the story. 



Mr* Jerry Ruble 

Peoria Association of Commerce 
General Manager 

We have all heard the statement. "School programs have little relation to the 'real world* and to the 
world of work'." This statement csn be cast aside by the concept of "career education." 

By virtue of a federal grant, our public school district was able to move into that exciting and 
revolutionary area of career education. The impact of this move in such a short period of time has been 
encouraging 

Listed are a number of spectre items that career education has brought about, and is attempting to 
bring about, in Peoria 

Following are the specifics: 

1 Career education has pulled the business community and the education community closer 
together 

2 When you start talking and working together, a learning process develops that can bridge "the 
understanding gap" that sometimes exists between education and the community. 
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3. The subjects of credibility and accountability in education are demanding much attention these 
days. Career education can assist the school districts in these areas, because the taxpayers and 
parents can relate and see accomplishments very readily through career education. 

4 The image of the school district has improved in the eyes of the community, and I feel that the 
image of our community has improved in the eyes of the teachers. 

5 Children now have a contact with adults other than their parents and teachers. The children are 
having an opportunity to develop additional dialogue with labor leaders and representatives of 
government and business. 

6. Career education has given children an opportunity to see people as they are and experience the 
community as it really is—a broadening of their horizons. 

As an example: A model career visit experience was developed by our school district and 
implemented by the Education Committee of the Peoria Association of Commerce. This career 
visit experience was centered toward people and not products and/or processes. Rather than a 
tour of their operation, businesses have agreed to provide one worker for every four to six students 
to conduct the interview. These interview trips into the community have been more meaningful 
and more productive educationally. Mis significant to note that the peoplebeing used as resource 
hosts have been from all levels, skilled and professional. 

7, Teachers have also become educated by bringing them into contact with people outside the 
school 

The above items came into being as a result of the activities listed below; 

1, On April 14, 1972, our school district and the Career Education Advisory Committee* in coopera- 
tion with personnel of the Illinois State Chamber of Commerce* invited fifteen hundred business 
leaders to attend a Career Education Seminar. 

2. In June 1972, the Peoria Association of Commerce assisted the Career Education Staff of our 
school district in planning and implementing the community resources portion of a two-week 
career education workshop tor 150 school district personnel. 

3 In August 1972, 300 business, government, and labor leaders were invited to a Career Education 
Workshop sponsored by the Peoria Association of Commerce. 

Out of this meeting a series of model career visits were developed. Community leaders were also 
given information concerning why and how their involvement in the schools with children is 
absolutely necessary. 

4 In September 1972, the Peoria Association of Commerce arranged for 400 teachers to interview 
labor, government, and business people at their respective locations. 

From September 1972* through February 1973, 6 months, there have been over 600 trips into the 
community which required bus transportation, It is impossible to estimate the number of children who 
have visited, for the first time, businesses within walking distance of their schools. The number of 
resource people that have been invited into classrooms, and the number of trips to resource people that 
have been taken utilizing private transportation is also impossible for me toestimate. Another reservoir 
of resource people that had been overlooked in the past were school personnel— custodians, nurses, 
secretaries, cafeteria workers, bur drivers, security guards, and maintenance men. 

The activities and programs that have been mentioned were not done prior to September 1972. These 
accomplishments were made because of additional staff, support personnel, and increased money for 
bus transportation. 
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If the federal or state career education monies are discontinued, some of the activities, suchasvisitsby 
resource people to the schools could be continued* However the support personnel and the availability 
of transportation would have to be curtailed. 

Our public school district has had financial difficulties the last few years, which has necessitated 
decreasing school personnel in general and support personnel in particular When people are 
removed, programs suffer We are hopeful that money will be provided so career education in Peoria 
will not suffer a most unfortunate and poorly timed death. It would be poorly timed because the 
taxpayers in Peoria, as well as throughout the country, are trying to understand this phenomenon into 
which today's education has developed 

The question can be asked. "Why should business people help improve education, and more specifi- 
cally augment career education?" The answer is simple. The future of their enterprise, the city of 
Peoria, and their nation depends on education. Next to the family, no single social institute does more 
than the education system to shape the lives of people. And people is what living is all about. 

Our children need and deserve career education. Weare committed to the development of a system that 
is responsive* and responsible, to the needs of society* of the economy* and the student. We believe 
these needs can best be met by the implementation of career education that will allow a student to 
explore various career opportunities, select the one that will provide the greatest rewards to both him 
and his eventual employer, and then to provide him with the specific tools needed to obtain a saleable 
skill in that area. 

To accomplish this educational goal will require the involvement and active participation of business 
and industry, labor unions, service organizations, government* school administrators, and school 
boards. 

Locally, we have tried some new approaches in the area of career education that did not work. We had 
to retrace our steps and then move on. However* I would rather try to do something and fail—than try to 
do nothing and succeed. This is a new concept for us. but we are learning. We are also excited by the 
possibilities. Young people do not need their parents 1 capital—only their parents 1 interest. Career 
education can bridge this gap also. 

John Gardner once stated* 'If a nation does* not respect its pfumbers as well as it s philosophers* neither 
its pipes or its theories will held water/' This we are learning in Peoria. 

In conclusion, let me remind you . . , there is little difference in people. But that little difference makes a 
big difference. The little difference is attitude. The big difference's whether it is positive or negative, fn 
Peona. our attitude toward career education is positive—thanks to the federal grant. 



Mrs, Irene E, Bohannon 
2702 West Kraute 

Teacher* Harrison Elementary School 

Peoria. Illinois, is deeply involved in a federally funded Career Awarenessand Exploration Program. As 
a teacher incorporating and implementing this program into my classroom, I feel it has been of great 
value. I have talked and worked with other teachers in District 150 who feel as strongly as f about the 
benefits of career education. I have been in Peoria two and one-half years, but have taught in other 
ichoot systems and this program would* I am sure, be welcomed in the other two areas of the country in 
which I h ,ive worked Many teachers who have been in Peoria for years have expressed to me that this 
program has been long overdue. 

This program has encompassed many different areas of education inside our schools, our immediate 
community, and our citv Through it has come greater communication among teachers in individual 
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schools and among schools in the district We have shared ideas verbally, as well as through the 
newspaper and newsletters. 

The initiation of this program has involved the community, and. of greater importance to me as a 
teacher parents in particular Our children have and are being exposed to an adult world of work. The 
students are gaining an understanding and a communication with adults not experienced heretofore. 

When the program was introduced, the community was quite receptive and we have constantly been 
using our community resources. 

Our students are benefiting by beinq able to interview all kinds of people in all kinds of positions. They 
have had opportunities to have "hi^ds-on" experiences. Until this program, they did not have such 
opportunities. 

As a teacher. I have tried to provide career awareness activities before the program* but was very 
limited. I had no guidelines to follow, no one with information and assistance, or finances with which to 
work. The program has enabled me as a teacher to follow through and tie in my career awareness with 
curriculum. I've taken advantage of the extra funds for bus services: the children have been able to 
relive their experiences becauseof access to films, slides, extra books in the library, and a multitude of 
other items* our school could not have otherwise afforded. The students and I could not have fully 
carried out many of our projects without access to the funds from the Career Awareness and Explora- 
tion grants. 

To be even more specific, without the Career Education staff to coordinate workshops and provide 
consultants this past summer and fall, the stipends offered, and the released time to be a part of 
workshops, the program could not have succeeded. We would not have been able to marshal publicity if 
it were not for these personnel being totally involved and speaking to community servlceorganizations. 

Resources that are available to me as a teacher are endless, Examples include a book on model career 
visits, a pamphlet on "People of Peoria," prepared forms for interviews, and questionnaires. We have 
our own handbook, lists of materials, and money to purchase them. We have continuous conatructive 
criticism and guidance from university consultants hired on a part-time9 month basis, coordinate, J 
services such as reading, library instructional materials and transportation to augment the program. 
Every library in the district hasa career education resourcecenterand at this time every school is in the 
process of building a community resource file. There are regularly scheduled career education inser- 
vice workshops once a month. These workshops are coordinated by the people that attended the 
summer workshops. This semester teachers are participating in a course held 2 nights a woek on career 
awareness credited through Western Illinois University. 

For the teachers it is a new experience to be asked to share our expertise with other teachers and this 
has encouraged a more creative approach to teaching and a professionalism in attitudes. We as 
teachers are beginning to realize that there are instructional resources at our fingeitips; for example, 
custodians, secretaries, school nurses, aides, and principals, They are being involved directly in the 
instructional program 

More importantly, the students are benefiting through the relevancy of the program. Education is 
becoming more meaningful to them. We are getting away from talking and we are doing. Things are 
becoming available to them. For instance, special experiences for disadvantaged children such as the 
train ride recently to Chillicothe. People are volunteering community services of all kinds. 

As an experienced teacher I truly feel that if this program isdiscontinuedwewill be doing our students a 
terrible injustice The career education program must have time to grow, As the business community 



No P(juipnn«nl ha* he**n purchased utilizing protect funds However several instructional materials companies have placed 
■Mtjipmont with ik on<i trial h,iM<; Encyclopedia Snlannica Corporation has placed over $25 000 worth of instructional material 
w nwnrf t r iO Uu 1 y*M' M no erst to th** district 
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becomes more aware of its benefits, Tm sure, they will, take more of the financial responsibility. This 
program is not just a school, teacher students, parent program, but one that will benefit the entire 
community. To discontinue the funding at this time would be a giant step backwards. It is for these 
reasons I strongly urge the continuation of the Career Awareness and Exploration Program. 



Mr Albert L Walker 

923 West Mlllman Street 

Principal, Webster Elementary School 

This is the first year Webster School has been involved with the implementation of the career education 
program. Webster School can best be described in the following manner: (1) school enrollment-290 
students. (2) racial com posit ion —86 percent black* 14 percent white. (3) socio-economic information— 
mostly low income, (4) building age— 75 years, and {5) grade organizat ion— prekindergarten-grade6. 

As a result of having career education, the students, staff, and community has been more involved in the 
total educational process. The staff-community relationship has been greatly improved. The overall 
resou cesof the school have increased as a direct result of our involvement But even more importantly, 
the students are motivated by the career education activities. By accomplishing the integration of 
career education into the regular academic program, schooling has taken on a new meaning for 
students. They are beginning to understand the maximum-learning with a purpose. 

I have noticed and teachers have expressed concern that many of the students have extremely low 
levels of personal aspiration. Many of usfeel low aspiration is simply a product of the students 1 feelings 
of inadequacy (i e . of low expectation for themselves). Activities carried out as a result of the career 
education program, especially increased contact with productive members of the community have 
definitely begun to modify students' level of aspiration in an upward direction Studentswho previously 
appeared tohavelittleaim in life have demonstrated by their communication and conduct that they are 
recognizing there is honor in work- They are also beginning to demonstrate an awareness that we live in 
a competitive world and that rewards can be gained if one is willing to give to get. For example, after 
visits to the community, and visits from community people into the school, students have been 
observed in the halls and in the classrooms discussing the question. "What can I be?" 

This program has added a measure of dignity to studf ntsand parents in the community. Many of the 
jobs held by residentsare of relatively tow status and meager pay. When these people are called upon to 
be educational resources to enhance programs for children they begin to recognize and appreciate 
their contributions and to feet that they are personally worthwhile. 

Although one can th ink of the great educational benefits afforded our children as a result of the career 
education program, one has only to consider what our academic programs would je like if we did not 
have career education. First many of the qualified individuals would not be available to provide the 
valuable information, coordination and support that has madt the career education program a 
success-career education staff, consultant help from local universities and from teachers representing 
other school districts, and members of the community. Secondly, the involvement of outside commu- 
nity business, parents and other individuals would be greatly reduced. Finally, many teachers* stu- 
dents, parents and community people would not be aware of the educationally related career activities 
so badly needed in our schools. (We have had inservice programs that started in April 1972 and have 
continued to the present.) 

Although the career education program is still young, it has enjoyed a great deal of success. I am 
concerned about what may happen it the financial support that made the program possible is discon- 
tinued too soon. My teachers and I are being asked to do things differently than we have done in the 
past It is not easy to change. It is important that needed support is supplied it old habits are to be 
broken and new techniques are to be learned and developed. 
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Even when financial support j$ available as it has been this year, restrictions on that support have 
hindered progress in the schools. For example, many of our teachers have indicated a desire to 
purchase articles that fall into the category of equipment. These are not particularly expensive articles 
and would add much toour instructional programs, but we havebeen told that it is not possible to utilize 
funds to purchase equipment. 

We firmly believe that careered ucat ion has a place in our curriculum by better educating and preparing 
our students for the "world of work." 



Dr Clyde R Smith 
1S01 West Bradley 

Parent, District 150 Volunteer and Career Education 
Professor-Consultant 

Career education in the elementary schools is, in my opinion, one of the most important curriculum 
innovations in education since the educational programs which resulted from the 1957 launching of the 
Russian Sputnik It is a privilege to be a part of career education, having begun the first such program 
on a volunteer basis with Glen Oak Elementary School in September 1970. For 2 years at Glen Oak we 
struggled with the procedures for making career education an integral part of the existing eighth grade 
curriculum. Due to lack of interest on the part of educators in general and those who control the 
economics of educational financing, we had to rely on all volunteer help from students, parents* and the 
community, Even with these great odds against us the program wasa success. Bee a use of the program 
at Glen Oak the National Vocational Guidance Association at its annual convention in Chicago on 
March 29, 1973. presented me with the 1972 Merit Award for outstanding contribution to the field of 
vocational guidance and career education. 

With the advent of the $250,000 grant from the Federal Govern mentcareereducation in Peoria'came of 
age" Now career education Isn't just for a few Every student in every grade in all 33 elementary schools 
has an opportunity to become aware of the tremendous number of opportunities available in the world 
of work. This program, while in its infancy, is bringing about a new understanding of the public shools 
on the part of teachers, students, parents, and the Peoria community as a whole. There is an enthusiasm 
which has not been seen in the classroom for some time. Peopleare talking about career education; not 
only in schools, but in churches, civic clubs, and fraternal organizations. 

The news media-radio, television, and newspaper have been behind this effort in a way hard for the 
average citizen to believe. I have had several spots on radio and television, including 2 hour-long radio 
programs, and two 1/2-hour television programs, with another 1/2-hour television program concerning 
career education in school District 150 on Sunday, March 18. Career Education has captu™d the 
imagination of thecommunit/ so much that, even before the airing oft he upcoming television special I 
have already been given another 1/2-hour television program for March 1974. 

Therefore, based on my personal experience, both as a volunteer for two years and as a professor- 
consultant under the federal contract, I can say that career education in the elementary schools of 
Peoria District 150 has been the most important educational innovation with which I have been involved 
in all my professional career It is my sincere hope that career education will receive continued support 
in Peoria and throughout Illinois. Seldom has a taxpayer received so great a return on such a small 
investment. 
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The key questions and related criterion questions you wrote for use in evaluating the evaluation 
activities reflected in the enclosed career guidance program evaluation report should relate to one or 
more of the following listed areas of concern. 



1 Purpose 
2< Objectives 

3, Personnel 

4. Design 

5 Instrumentation 



6. Information sources 

7, Data 

8, Findings 

9. Dissemination 
10. Costs 



In addition, each of your criterion questions should relate to and assist in answering its respective key 
question 



GROUP r? fry^ 

ACTIVITY-. V ^ "'\<\\T\Y*X^ 



Note: The following outline is to be used by the workshop facilitator. 



Facilitator s Outline 


Notes 


A Getting Started 




1 Inforn, participants that as a group 
they will develop an evaluation plan 
tc evaluate the processes and tech- 
niques used in the formative evalua- 
tion of the careerguidance program 
reflected in the evaluation report 
included with this learning experi- 
ence 




2 Ask the participants to review the 
reading for Competency 7 on page 
16 and the evaluation report used in 
thn Individual Activity 
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Facilitator 4 * Outline 



Notes 



B< Develop Evaluation Plan 

1 Explain that the participants are to 
assume the role of the career guid- 
ance personnel whose program has 
recently been evaluated. The report 
of that evaluation is to serve as the 
focus of their efforts to develop a 
plan for evaluating this evaluation of 
"their" career guidance program. 

2 Explain that the evaluation plan 
should include the following 
components: 

• Purpose statement 

• Personnel involved 

• Timetable for the activity 

• Areas of concern and evaluation 
qucstior > 

• Instrumentation 

• Data collection 

• Data analysis 

• Reporting results 

• Utilizing results 

3. Have participants read the evalua- 
tion report from the Individual 
Activity. 

4. Have them* as a group, generate an 
outline of what should be presented 
in each component of the evaluation 
plan. 

5. After the outline has been devel- 
oped, assign each component to 
individuals or small groups and ask 
them to "flesh out" the outline for 
their component. 

6 Allow time for all to complete the 
above activity. 

C< Report on the Evaluation Plan 

1 Ask participants to present to the 
total group their assigned compo- 
nent of the evaluation plan. 

2 Discuss the viability of each com- 
ponent as presented and its rela- 
tionship to other components and 
the entire plan 

68 



Project or write the components so they can be 
seen by alt participants, and from the discus- 
sion develop the outline under each component, 
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EVALUATION 



PARTICIPANT SELF-ASSESSMENT QUESTIONNAIRE 

I Narrm (Ophonall 3 Date _ 

7 Position Title 4 Module Number 

Agency Setting (Circle the appropriate number) 



6 Flenienlary School 

7 Secondary School 

8 Posisecondary School 
0 Collrg** University 



10 JTPA 14 

11 Veterans IS 

12 Church 

13 Corrections 16 

17 



Youth Services 
Busines' Industry 
Management 
Business/Industry Labor 
Parent Group 



18 Municipal Office 

19 Service Organization 

20 Slate Government 

21 Other 



Workshop Topics 



PREWORKSHOP NEED FOR TRAIN- 
ING Degree of Weed (circle one for 
each workshop topic) 



POSTWORKSHOP MASTERY OF 
TOPICS Degree of Mastery (circle 
one for each workshop topic) 



///// 



///> 



/ 



1 Writing key evaluation questions to be 
answered in the assessment of client 
performance objectives for selected 
program components or areas of con- 
cern in the evaluation. 01234 01234 

2 Writing key evaluation quest ons to be 
answered in the assessment < program 
process objectives for selected areas of 

concern in the evaluation. 01234 01234 

3 Writing criterion questions and related 
it'>ms for an instrument which will be 
useful in assessing the attainment Of 

program process objectives, 01234 01234 

4 Adapting existing measurement instru- 
ments for use in assessing the attain- 
ment of program process objectives for 

your career guidance program, 01234 01234 

5 Writing key evaluation questions and 
related criterion questions to be an- 
swered in evaluating an evaluation 

process 01234 01234 

6 Drafting an evaluation plan for assess* 

ing an evaluation activity. 01234 01234 

7 Describing hypothetical applications of 
common experimental designs to 
determine if client gains can be attrib* 
uted to a particular career guidance 

program activity or service, 01234 01234 
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PREWORKSHOPNECO FOR TRAIN* postworkshop mastery of 
ING Degtee ot Need |c*i c\e one tor TOPICS 009/00 0/ Mastery (circle 
Workshop ToplCI each workshop loptci one tor each workshop topic) 

///// ///// 



8 Designing an evaluation plan that will 
demonstrate the impact that a selected 
program component has on intended 
gains made by clients. 0 

9. Analyzing and displaying evaluation data 

in usable form. 0 

10. Administering evaluation instruments 

according to the evaluation plan, 0 



Overall Assessment on Topic of Evaluate 
Guidance Activities 0 

Comments; 



12 3 4 

12 3 4 

12 3 4 

12 3 4 



0 12 3 4 

0 12 3 4 

0 12 3 4 

0 12 3 4 
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Trainer's Assessment Questionnaire 



Trainer 

Title of Module 



Date: 



Module Number: 



Training Time to Complete Workshop: 

Participant Characteristics 

Numberin Group . Number of Males 

Distribution by Position 

-_ Elementary School 

Secondary School 
. . Postsecondary School 
, College/University 
. JTPA 

-_. Veterans 

. Church 
Corrections 



hrs. 



mia 



Number of Females. 



Youth Services 

Business/Industry Management 

Business/Industry Labor 

Parent Group 

Municipal Office 

Service Organization 

State Government 

Other 



PART I 

WORKSHOP CHARACTERISTICS— Instruction*; Please provide any comments on the methods and 
materials used, both those contained in the module and others that are not listed. Also provide any 
comments concerning your overall reaction to the materials, learners 1 participation or any other 
positive or negative factors that could have affected the achievement of the module's purpose. 

1 Methods: (Compare to those suggested in Facilitator's Outline) 



2 Materials (Compare to those suggested in Facilitator's Outline) 



3 Reaction (Participant reaction to content and activities) 
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PART II 

WORKSHOP IMPACT— Instruction*: Use Performance Indicators to judge degree oi mastery. (Com- 
plete responses tor all activities. Those that you did not teach would receive 0 ) 

Group's Degree of Mastery 





Not 


Little 


Some 


Good 


Outstanding 




Taught 








{over 75%) 




Note: Circle the number that beat reflects your opinion of 


group maatery* 


Learning Experience 1 












Group 


A 
U 




* 


*» 
O 




Individual 


0 




2 


3 




Learning Experience 2 












Group 


0 




2 


3 


4 


Individual 


0 




2 


3 




Learning Experience 3 












Group 


0 




2 


3 




Individual 


0 




2 


3 




Learning Experience 4 












Group 


0 




2 


3 




Individual 


0 




2 


3 




Learning Experience 5 












Group 


0 




2 


3 




Individual 


0 




2 


3 





Code: 

Little; With no concern for lime or circumstances within training setting if it appears that lessthan 25% ollhe learners 
achieved what was intended to be achieved 

Some: With no concern for time or circumstances within th* (raining setting if il appears I hat less thdncios* to halt of 
trw learners achieved the learning experience 

Good: With no concern for tim rt or circumstances within thetraining setting if rt appears that 50%-75^> have achieved 
as expected 

Outstanding; If more than 75% of learners mastered the content as expected 
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PART III 



SUMMARY DATA SHEET— Inductions: In order to gain an overall idea as to mastery impact 
achieved across the Learning Experiences taught, complete the following tabulation. Transfer the 
number tor the degree of mastery on each Learning Experience (i.e., group and individual) from the 
Workshop Impact form to the columns below. Add the subtotals to obtain your total module score. 



GROUP 

Learning Experience 

1 score (1-4) 

2 score (1-4) 

3 score (1-4) 

4 score (1-4) 

5 ■■ score (1-4) 
Total 

(add up) 



INDIVIDUAL 

Learning Experience 

1 = score (1-4) 

2 = score (1-4) 

3 = score (1-4) 

4 = score (1-4) 

5 = score (1-4) 

Total 
(add up) 



Total of the GROUP learning experience scores and INDIVIDUAL learning experience scores = 
. Actual Total Score Compared to Maximum Total' 

'Maximum total is the number of learning experiences taught times four (4). 
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Performance Indicators 

As you conduct the workshop component of this training module >the facilitator's outline will suggest 
individual or group activities that require written or oral responses. The following list ol performance 
indicators will assist you in assessing the quality of the participants' work: 

Module Title Evaluate Guidance Activities 

Module Number; CG E-1 

Group Learning Activity Performance Indicators to Be Used tor Learner Assessment 



Group Activity Number 1: 

Develop program process-oriented 
key questions, 



1. Participants were able to identify program areas ol concern, 

2 Participants understood thecriteriaforevaluating key questions. 

3. Participants were able to develop key questions related to the 
criteria. 



Group Activity Number 2: 

Design a portion of an evaluation 
plan 



1. Participants understood the experimental design in a specific 
evaluation plan. 



Group Activity Number 3: 

Adapt a given instrument tor use 
in evaluating a career guidance 
program. 



1, Participants were able to modify the given instrument. 

2 Participants were able to justify the reasons tor their actions 
related to each item. 



Group Activity Number 4; 

Participate in a role-playing situ- 1. Participants were able to complete the roles assigned to them, 
ation in which evaluation instru- 
ments are administered 



Group Activity Number 5: 

Develop a plan to evaluate the tor- 1 Participants were able to generate an outline of an evaluation 
mativeevaluation ofa careerguid- report with all of the components, 
ance program 
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ADDITIONAL RESOURCES 



Materials listed in this resource section are ones that can provide you with additional knowledge lor 
improving your career guidance program through complete and effective evaluation. There were 
hundreds ol good reference materir Is on program evaluation but those listed below are a few that 
seemed to be most useful especially k onsidering the module competencies. Ideas from some of these 
resource materials were used in the module itself. 



Conducing SummaOve Evaluation. Jean Wolman, 
National Consortium on Competency Based Staff 
Development. American Institutes for Research, 
P O Box 1 113, Palo Alto* California 94302. 1978, 

This publication is designed to increase the read* 
er s understanding of the importance of summa- 
tive evaluation and the basic skills needed to 
design and carry out an evaluation. The focus of 
the document is—Did the program meet its goals? 
The topics covered include: designing a summa- 
tive evaluation, selecting/developing the instru- 
ments, administering the instruments, processing 
and analyzing the data* relating program costs to 
program effects, and making decisions based on 
results. Six activities placed throughout the mod- 
ule provide the reader with an opportunity to 
practice the concepts presented within the mate- 
rial The mformation can be adapted to any 
situation. 

Evaluation m Education: A Practitioner's Guide* 
J M GottmanandR E Clasea F.E> Peacock Pub- 
lishers, Inc., 401 West Irving Park Road, Itasca, 
Ilhnois60143, 1972, 

This document uses a workbook approach in 
presenting the basics of educational evaluation. 
It seeks to develop skills in four areas: needs 
assessment, developing objectives and measure- 
ment procedures, flow charting, and designing 
and using quality control (evaluation) procedures. 
This publication relates to numerous competen- 
cies presented in the evaluation module. 



Evaluation Research: Methods of Assessing Pro- 
gram Effectiveness. Carol K Weiss, Prentice Hall, 
Englewood Cliffs* New Jersey, 1972, 

This publication is a comprehensive, concise 
look at evaluation in the area of social programs, 
A practical emphasis is placed on evaluation in 
the context of action programs— e,g,, schools* 
mental health institutions* and job training pro- 
grams, It covers formulating goals, unanticipated 
consequences, measurement variables* data col- 
lection, evaluation design, cost-benefit analysis 
and Planning-Programming-Budgeting-System* 
problems in the action setting, and use of results. 
This document is helpful for someone desiring an 
in-depth look at evaluation. 

Implementing Career Education: Evaluation Proj- 
ect. Or, Arland Benson, Roseville Area Schools. 
Independent School Oistrict #623, 1251 W. County 
Road B-2< Roseville* Minnesota 55113. 

The purpose of this booklet is to provide concrete 
examples of how to evaluate a comprehensive 
careereducation program. Theapproach devalu- 
ation presented is one of a plying logic and 
objectivity to the process of judging the merits of 
career education activities. The contents of the 
publication address the unique considerations in 
career education activities, needs assessments, 
various types of instruments, ev* luation designs* 
data collection, unintended effects, cost- 
effectiveness, and reporting results, The docu- 
ment is strongly school focused, but may be use- 
ful to others. 
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KEY PROJECT STAFF 

The Competency- Based Career Guidance Module Series was developed by a consortium of agencies. The 
following list represents key staff in each agency that worked on the project over a five-year period. 



The Nations) Center for Research In Vocational Education 

Harry N Drier Consortium Director 

Robert E Campbell Project Director 

Linda A ptister Former Project Director 

Robert Bhaerman Research Specialist 

Karen Klmmel Boyle Program Associate 

Fred Williams Program Associate 

American Institute* for Reseaicfc 

0 Brian Jones Project Director 

Linda Phillips- Jones Associate Project Director 

Jack Hamilton Associate Project Director 

University oi MHseourt-CoiuaiMa 



American Association for Counseling and Development 

Jsne Howard Jasper Former Project Director 

American Vocational Association 

Wayne LeRoy Former Project Director 

Ron! Posner Former Project Director 

U.S. Department of tducsllon, Office of AduH 



David *rltchard Project Officer 

Holli Condon Project Officer 



Norman C Oysbers Project Director 



A number of national loaders representing a variety of agencies and organizations added their expertise to the 
project as members of national panels of experts. These leaders were*- 



Ms Giaoa Swinger 
Pesl President 
National Parent-Teacher 
Assoc wlton 

Or Frank Sows 

Former Executive Director 

Ms Jene Razeghi 
Education Coordinator 
American Coalition of Citizens 
with Disabilities 

Mr Robert I Craig 
Vice President 

Government and Public Allure 
American Society tor Training 
and Development 

Dr Walter Davis 
Director ol Education 
AfL CIO 

Dr Richard OiEugemo 
Senior i: tf *^hve Associate 
(representing Congressman Bill 

Goodlmg) 
House Education and labor 

Committee 

Mr Oscar Giernes 
Administrator (Retired} 
US Department ot labor 
Division ol Employment and 
Training 

Dr Robert W Glove* 
Director and Chairperson 
Federal Committee on 

Apprenticeship 
The University ot Texas at Austin 



Dr. Jo Haysllp 
Director ol Planning and 

Development In Vocational 

Rehabilitation 
New Hampshire Stele Department 

of Education 

Mrs Madeleine Hemming* 
National Alliance for Business 

Dr Edwin Herr 
Counselor Educator 
Pennsylvania Stale University 

Dr Elaine House 
Professor Emeritus 
Rutgers University 

Dr DavtdLacey 
Vice President 

Personnel Planning and Business 

Integration 
CIGNA Corporation 

Dr Howard A Matthews 
Assistant Slatt Director 
Education (representing Senator 

DnnG Hatch) 
Committee on Labor and Human 

Resources 

Dr leeMcMurrin 
Superintendent 
Milwaukee Public Schools 

Ms Nenine Meiklefohn 
Assistant Director ot Legislation 
American Federation ot State. 
County, and Municipal Employees 



Dr Joseph o. Mills 

State Director ol Vocational 

Education 
Florida Department of Education 

Or Jack Myers 

Director ot Health Policy Study and 

Private Sector Initiative Study 
American Enterprise Institute 

Mr Raid Rundelt 

Director of Personnel Development 
General Motors Corporation 

Mrs. Dorothy Shields 
Education 

American Federation of Labor/ 
Congress ol Industrial 
Organizations 

Dr Ldrba re Thompson 
Former State Superintendent 
Wisconsin Department ot Public 
Instruction 

Ms Joan wills 
Director 

Employ men I and Training Division 
National Governors' Association 

Honorable Chalmers P Wylia 

Congressman/Ohio 

U S Congress 
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